[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:00 dist-test-slave-mrw4 krb5kdc[2712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:00 dist-test-slave-mrw4 krb5kdc[2712](info): set up 2 sockets
Sep 26 19:39:00 dist-test-slave-mrw4 krb5kdc[2712](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 19:39:01.722000 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:01 dist-test-slave-mrw4 krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:39:01.870808 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35101
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:35101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:02.163803 2725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:02.164069 2725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:02.164201 2725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:02.171128 2725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:02.171232 2725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:02.171306 2725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:02.171381 2725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:02.171440 2725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:02.180622 2725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:35101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:02.181846 2725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:02.183032 2725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:02.194175 2730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:02.194123 2733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:02.194151 2731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:02.194417 2725 server_base.cc:1034] running on GCE node
I20260926 19:39:02.195358 2725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:02.196691 2725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:02.197912 2725 hybrid_clock.cc:648] HybridClock initialized: now 1790451542197849 us; error 45 us; skew 500 ppm
I20260926 19:39:02.198377 2725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:02 dist-test-slave-mrw4 krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:02.201324 2725 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:39:02.202705 2725 webserver.cc:460] Webserver started at http://127.2.164.126:38555/ using document root <none> and password file <none>
I20260926 19:39:02.203342 2725 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:02.203461 2725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:02.203737 2725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:02.206671 2725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "3d35886e49e840879413e6c20e1d7be0"
format_stamp: "Formatted at 2026-09-26 19:39:02 on dist-test-slave-mrw4"
server_key: "008a87792cba914802a3852c6e730911"
server_key_iv: "aa4ab14e253f91aa4c20de23469c5ab0"
server_key_version: "encryptionkey@0"
I20260926 19:39:02.207367 2725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "3d35886e49e840879413e6c20e1d7be0"
format_stamp: "Formatted at 2026-09-26 19:39:02 on dist-test-slave-mrw4"
server_key: "008a87792cba914802a3852c6e730911"
server_key_iv: "aa4ab14e253f91aa4c20de23469c5ab0"
server_key_version: "encryptionkey@0"
I20260926 19:39:02.212385 2725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:39:02.215818 2740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:02.217151 2725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:39:02.217329 2725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "3d35886e49e840879413e6c20e1d7be0"
format_stamp: "Formatted at 2026-09-26 19:39:02 on dist-test-slave-mrw4"
server_key: "008a87792cba914802a3852c6e730911"
server_key_iv: "aa4ab14e253f91aa4c20de23469c5ab0"
server_key_version: "encryptionkey@0"
I20260926 19:39:02.217490 2725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:02.235992 2725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:02.240124 2725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:02.240330 2725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:02.256276 2725 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:35101
I20260926 19:39:02.256294 2791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:35101 every 8 connection(s)
I20260926 19:39:02.257534 2725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:02.261039 2792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:02.263432 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 2725
I20260926 19:39:02.263688 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:02.265412 2705 external_mini_cluster.cc:1428] Setting key 2aa0ad530690bb622889af064459233b
I20260926 19:39:02.271589 2792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0: Bootstrap starting.
I20260926 19:39:02.274420 2792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:02.275319 2792 log.cc:824] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:02.277817 2792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0: No bootstrap required, opened a new log
I20260926 19:39:02.284749 2792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35886e49e840879413e6c20e1d7be0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35101 } }
I20260926 19:39:02.285060 2792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:02.285140 2792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d35886e49e840879413e6c20e1d7be0, State: Initialized, Role: FOLLOWER
I20260926 19:39:02.285745 2792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35886e49e840879413e6c20e1d7be0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35101 } }
I20260926 19:39:02.285972 2792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:02.286130 2792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:02.286288 2792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:02.288384 2792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35886e49e840879413e6c20e1d7be0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35101 } }
I20260926 19:39:02.288832 2792 leader_election.cc:302] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d35886e49e840879413e6c20e1d7be0; no voters:
I20260926 19:39:02.289345 2792 leader_election.cc:288] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:02.289475 2797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [term 1 FOLLOWER]: Leader election won for term 1
Sep 26 19:39:02 dist-test-slave-mrw4 krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:02.291078 2792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:02.292062 2797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [term 1 LEADER]: Becoming Leader. State: Replica: 3d35886e49e840879413e6c20e1d7be0, State: Running, Role: LEADER
I20260926 19:39:02.292629 2797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35886e49e840879413e6c20e1d7be0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35101 } }
I20260926 19:39:02.295550 2799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d35886e49e840879413e6c20e1d7be0. Latest consensus state: current_term: 1 leader_uuid: "3d35886e49e840879413e6c20e1d7be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35886e49e840879413e6c20e1d7be0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35101 } } }
I20260926 19:39:02.295893 2799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:02.296767 2798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d35886e49e840879413e6c20e1d7be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d35886e49e840879413e6c20e1d7be0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35101 } } }
I20260926 19:39:02.297039 2798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:02.303704 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:02.268920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.2.164.126:35101)
0926 19:39:02.269652 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:02.269663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:02.270759 (+ 1096us) server_negotiation.cc:407] Connection header received
0926 19:39:02.279064 (+ 8305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:02.279082 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:02.279425 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:02.279758 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:02.282398 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:02.283675 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:02.285221 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:02.285861 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:02.286028 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:02.291979 (+ 5951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:02.291992 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:02.292011 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:02.292076 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:02.296412 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:02.297377 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:02.297387 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:02.297395 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:02.297618 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:02.298238 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:02.298244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:02.298247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:02.299039 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:02.299345 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:02.301970 (+ 2625us) server_negotiation.cc:299] Negotiation successful
0926 19:39:02.302375 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":337,"threads_started":1}
I20260926 19:39:02.309528 2807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:02.314388 2807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:02.322540 2807 catalog_manager.cc:1347] Generated new cluster ID: 96b7ac5471fe42faac2c1c2ce5dba849
I20260926 19:39:02.322642 2807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:02.348908 2807 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:02.350747 2807 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:02.362010 2807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d35886e49e840879413e6c20e1d7be0: Generated new TSK 0
I20260926 19:39:02.362974 2807 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:39:02.404454 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35101
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:02.705663 2818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:02.705962 2818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:02.706100 2818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:02.713310 2818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:02.713421 2818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:02.713569 2818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:02.723186 2818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:02.724391 2818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:02.725708 2818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:02.737829 2824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:02.737862 2826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:02.737924 2823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:02.738394 2818 server_base.cc:1034] running on GCE node
I20260926 19:39:02.738946 2818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:02.739766 2818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:02.740952 2818 hybrid_clock.cc:648] HybridClock initialized: now 1790451542740907 us; error 38 us; skew 500 ppm
I20260926 19:39:02.741269 2818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:02 dist-test-slave-mrw4 krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:02.744055 2818 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:39:02.745334 2818 webserver.cc:460] Webserver started at http://127.2.164.65:43025/ using document root <none> and password file <none>
I20260926 19:39:02.745954 2818 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:02.746068 2818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:02.746345 2818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:02.748997 2818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "1d91220b15184ddfa3a3166cbc6d2642"
format_stamp: "Formatted at 2026-09-26 19:39:02 on dist-test-slave-mrw4"
server_key: "08a70cf4358f11c2113fb2fa15a742fd"
server_key_iv: "d6e7b5c73c482076b12b2f3c89700c22"
server_key_version: "encryptionkey@0"
I20260926 19:39:02.749778 2818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "1d91220b15184ddfa3a3166cbc6d2642"
format_stamp: "Formatted at 2026-09-26 19:39:02 on dist-test-slave-mrw4"
server_key: "08a70cf4358f11c2113fb2fa15a742fd"
server_key_iv: "d6e7b5c73c482076b12b2f3c89700c22"
server_key_version: "encryptionkey@0"
I20260926 19:39:02.754539 2818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 19:39:02.757733 2833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:02.758975 2818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:39:02.759107 2818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "1d91220b15184ddfa3a3166cbc6d2642"
format_stamp: "Formatted at 2026-09-26 19:39:02 on dist-test-slave-mrw4"
server_key: "08a70cf4358f11c2113fb2fa15a742fd"
server_key_iv: "d6e7b5c73c482076b12b2f3c89700c22"
server_key_version: "encryptionkey@0"
I20260926 19:39:02.759270 2818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:02.781734 2818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:02.785113 2818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:02.785362 2818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:02.786209 2818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:02.787397 2818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:02.787470 2818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:02.787588 2818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:02.787643 2818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:02.817024 2818 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:37261
I20260926 19:39:02.817099 2945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:37261 every 8 connection(s)
I20260926 19:39:02.818336 2818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:02.828740 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 2818
I20260926 19:39:02.828964 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:02.829736 2705 external_mini_cluster.cc:1428] Setting key 228d26de1fa53be83b1598d03f8d68d7
Sep 26 19:39:02 dist-test-slave-mrw4 krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451542, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:02.836587 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:02.820524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:42641 (local address 127.2.164.126:35101)
0926 19:39:02.820878 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:02.820883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:02.821862 (+ 979us) server_negotiation.cc:407] Connection header received
0926 19:39:02.823585 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:02.823591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:02.823666 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:02.823758 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:02.825197 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:02.825902 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:02.827337 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:02.827738 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:02.827844 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:02.831514 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:02.831520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:02.831525 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:02.831562 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:02.833894 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:02.834546 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:02.834550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:02.834553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:02.834660 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:02.835029 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:02.835034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:02.835038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:02.835298 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:02.835421 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:02.836047 (+ 626us) server_negotiation.cc:299] Negotiation successful
0926 19:39:02.836242 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":180,"threads_started":1}
I20260926 19:39:02.837514 2949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:02.821004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35101 (local address 127.2.164.65:42641)
0926 19:39:02.821712 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:02.821754 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:02.823338 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:02.824000 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:02.824009 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:02.824493 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:02.825026 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:02.825043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:02.826033 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:02.826036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:02.827208 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:02.827220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:02.827935 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:02.827941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:02.828162 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:02.828770 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:02.828798 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:02.831289 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:02.834026 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:02.834034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:02.834050 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:02.834435 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:02.834755 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:02.834759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:02.834763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:02.834932 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:02.835411 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:02.835418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:02.835718 (+ 300us) client_negotiation.cc:769] Sending connection context
0926 19:39:02.835943 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 19:39:02.836427 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":289,"threads_started":1}
I20260926 19:39:02.838852 2946 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35101
I20260926 19:39:02.839159 2946 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:02.840068 2946 heartbeater.cc:511] Master 127.2.164.126:35101 requested a full tablet report, sending...
I20260926 19:39:02.842542 2757 ts_manager.cc:194] Registered new tserver with Master: 1d91220b15184ddfa3a3166cbc6d2642 (127.2.164.65:37261)
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260926 19:39:02.844229 2757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:42641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:39:02.863931 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35101
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:03.157712 2952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:03.157972 2952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:03.158069 2952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:03.164915 2952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:03.165009 2952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:03.165123 2952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:03.174235 2952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:03.175341 2952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:03.176525 2952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:03.188609 2960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:03.188649 2958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:03.188670 2957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:03.194802 2952 server_base.cc:1034] running on GCE node
I20260926 19:39:03.195395 2952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:03.196277 2952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:03.197503 2952 hybrid_clock.cc:648] HybridClock initialized: now 1790451543197451 us; error 43 us; skew 500 ppm
I20260926 19:39:03.197805 2952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:03 dist-test-slave-mrw4 krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451543, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:03.200742 2952 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:39:03.202153 2952 webserver.cc:460] Webserver started at http://127.2.164.66:36763/ using document root <none> and password file <none>
I20260926 19:39:03.202800 2952 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:03.202906 2952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:03.203178 2952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:03.206001 2952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "178e062150a44cf1a953d4fdb799b63d"
format_stamp: "Formatted at 2026-09-26 19:39:03 on dist-test-slave-mrw4"
server_key: "2a3113f6db0443d7e59b4f109cf5a1f3"
server_key_iv: "3d40266d70adbb4465b4e7d0d31986f6"
server_key_version: "encryptionkey@0"
I20260926 19:39:03.206699 2952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "178e062150a44cf1a953d4fdb799b63d"
format_stamp: "Formatted at 2026-09-26 19:39:03 on dist-test-slave-mrw4"
server_key: "2a3113f6db0443d7e59b4f109cf5a1f3"
server_key_iv: "3d40266d70adbb4465b4e7d0d31986f6"
server_key_version: "encryptionkey@0"
I20260926 19:39:03.211728 2952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:39:03.215895 2967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:03.217314 2952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 19:39:03.217437 2952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "178e062150a44cf1a953d4fdb799b63d"
format_stamp: "Formatted at 2026-09-26 19:39:03 on dist-test-slave-mrw4"
server_key: "2a3113f6db0443d7e59b4f109cf5a1f3"
server_key_iv: "3d40266d70adbb4465b4e7d0d31986f6"
server_key_version: "encryptionkey@0"
I20260926 19:39:03.217589 2952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:03.231972 2952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:03.235421 2952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:03.235627 2952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:03.236394 2952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:03.237564 2952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:03.237629 2952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:03.237722 2952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:03.237763 2952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:03.266117 2952 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:45753
I20260926 19:39:03.266129 3079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:45753 every 8 connection(s)
I20260926 19:39:03.267519 2952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:03.276197 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 2952
I20260926 19:39:03.276381 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:03.276885 2705 external_mini_cluster.cc:1428] Setting key 001b39dcf12e69fdcfb1653ab6df8bd9
Sep 26 19:39:03 dist-test-slave-mrw4 krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451543, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:03.286255 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:03.269922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:45937 (local address 127.2.164.126:35101)
0926 19:39:03.270116 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:03.270122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:03.271223 (+ 1101us) server_negotiation.cc:407] Connection header received
0926 19:39:03.272944 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:03.272950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:03.273026 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:03.273146 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:03.274479 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.275223 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.277149 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.277587 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.277681 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:03.281065 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:03.281069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:03.281072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:03.281101 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:03.283538 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.284181 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.284186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.284188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.284307 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.284695 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.284700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.284703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.284941 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:03.285064 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:03.285736 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 19:39:03.285955 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:39:03.287168 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:03.270335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35101 (local address 127.2.164.66:45937)
0926 19:39:03.271060 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:03.271108 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:03.272680 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:03.273337 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:03.273345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:03.273811 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:03.274326 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:03.274341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.275343 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.275347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:03.277015 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:03.277030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.277782 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.277789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:03.278022 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:03.278660 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:03.278682 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:03.280876 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:03.283660 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:03.283668 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:03.283683 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:03.284075 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:03.284412 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:03.284417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:03.284421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:03.284596 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:03.285057 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:03.285064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:03.285414 (+ 350us) client_negotiation.cc:769] Sending connection context
0926 19:39:03.285639 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 19:39:03.286130 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":289,"threads_started":1}
I20260926 19:39:03.288614 3080 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35101
I20260926 19:39:03.288892 3080 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:03.289584 3080 heartbeater.cc:511] Master 127.2.164.126:35101 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260926 19:39:03.290866 2757 ts_manager.cc:194] Registered new tserver with Master: 178e062150a44cf1a953d4fdb799b63d (127.2.164.66:45753)
I20260926 19:39:03.291647 2757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:45937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:39:03.309612 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35101
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:03.607414 3085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:03.607666 3085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:03.607785 3085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:03.614576 3085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:03.614683 3085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:03.614830 3085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:03.623997 3085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:03.625101 3085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:03.626323 3085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:03.638141 3090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:03.638108 3091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:03.638170 3093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:03.638903 3085 server_base.cc:1034] running on GCE node
I20260926 19:39:03.639400 3085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:03.640175 3085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:03.641359 3085 hybrid_clock.cc:648] HybridClock initialized: now 1790451543641311 us; error 40 us; skew 500 ppm
I20260926 19:39:03.641623 3085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:03 dist-test-slave-mrw4 krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451543, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:03.644446 3085 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:39:03.645795 3085 webserver.cc:460] Webserver started at http://127.2.164.67:40731/ using document root <none> and password file <none>
I20260926 19:39:03.646418 3085 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:03.646523 3085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:03.646790 3085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:03.649415 3085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "76b0173381bb435a91c9ea8f782b0dd9"
format_stamp: "Formatted at 2026-09-26 19:39:03 on dist-test-slave-mrw4"
server_key: "bd34d9dd175ffeeefb5253e746b91228"
server_key_iv: "2ffecc9b7db7bb4721c74e0b13d2df75"
server_key_version: "encryptionkey@0"
I20260926 19:39:03.650097 3085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "76b0173381bb435a91c9ea8f782b0dd9"
format_stamp: "Formatted at 2026-09-26 19:39:03 on dist-test-slave-mrw4"
server_key: "bd34d9dd175ffeeefb5253e746b91228"
server_key_iv: "2ffecc9b7db7bb4721c74e0b13d2df75"
server_key_version: "encryptionkey@0"
I20260926 19:39:03.654702 3085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:39:03.657699 3100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:03.658820 3085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:39:03.658952 3085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "76b0173381bb435a91c9ea8f782b0dd9"
format_stamp: "Formatted at 2026-09-26 19:39:03 on dist-test-slave-mrw4"
server_key: "bd34d9dd175ffeeefb5253e746b91228"
server_key_iv: "2ffecc9b7db7bb4721c74e0b13d2df75"
server_key_version: "encryptionkey@0"
I20260926 19:39:03.659113 3085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:03.680070 3085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:03.683538 3085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:03.683769 3085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:03.684607 3085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:03.685772 3085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:03.685845 3085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:03.685966 3085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:03.686024 3085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:03.719506 3085 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:39185
I20260926 19:39:03.719548 3212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:39185 every 8 connection(s)
I20260926 19:39:03.720714 3085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:03.721931 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 3085
I20260926 19:39:03.722113 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:03.722514 2705 external_mini_cluster.cc:1428] Setting key 971ef3f73d75d4c4d17879cd6c933802
Sep 26 19:39:03 dist-test-slave-mrw4 krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451543, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:03.738843 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:03.723160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:58485 (local address 127.2.164.126:35101)
0926 19:39:03.723344 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:03.723349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:03.724513 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 19:39:03.726364 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:03.726368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:03.726429 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:03.726504 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:03.727793 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.728500 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.730031 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.730387 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.730473 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:03.733587 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:03.733592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:03.733595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:03.733624 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:03.735891 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.736601 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.736606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.736609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.736737 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.737367 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.737371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.737374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.737652 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:03.737774 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:03.738449 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 19:39:03.738614 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:39:03.739653 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:03.723630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35101 (local address 127.2.164.67:58485)
0926 19:39:03.724348 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:03.724396 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:03.726130 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:03.726682 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:03.726689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:03.727141 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:03.727675 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:03.727695 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.728652 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.728657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:03.729918 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:03.729930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.730547 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.730551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:03.730737 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:03.731348 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:03.731377 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:03.733412 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:03.736022 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:03.736030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:03.736044 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:03.736495 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:03.736924 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:03.736929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:03.736932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:03.737157 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:03.737755 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:03.737761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:03.738116 (+ 355us) client_negotiation.cc:769] Sending connection context
0926 19:39:03.738330 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 19:39:03.738666 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":285,"threads_started":1}
I20260926 19:39:03.740924 3213 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35101
I20260926 19:39:03.741158 3213 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:03.741806 3213 heartbeater.cc:511] Master 127.2.164.126:35101 requested a full tablet report, sending...
I20260926 19:39:03.743057 2757 ts_manager.cc:194] Registered new tserver with Master: 76b0173381bb435a91c9ea8f782b0dd9 (127.2.164.67:39185)
I20260926 19:39:03.743783 2757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:58485
I20260926 19:39:03.749909 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:39:03 dist-test-slave-mrw4 krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451543, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 19:39:03 dist-test-slave-mrw4 krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451543, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:03.789247 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:03.775045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.2.164.65:37261)
0926 19:39:03.775608 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:03.775618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:03.775696 (+ 78us) server_negotiation.cc:407] Connection header received
0926 19:39:03.775889 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:03.775896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:03.776118 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:03.776253 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:03.777657 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.778637 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.779868 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.780405 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.780519 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:03.784154 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:03.784164 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:03.784179 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:03.784219 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:03.786678 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.787261 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.787268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.787270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.787390 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.787751 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.787755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.787757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.788086 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:03.788364 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:03.788650 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 19:39:03.788894 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":244,"threads_started":1}
W20260926 19:39:03.790408 2861 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56166
Sep 26 19:39:03 dist-test-slave-mrw4 krb5kdc[2712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 19:39:03 dist-test-slave-mrw4 krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:03.829730 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:03.815642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56176 (local address 127.2.164.65:37261)
0926 19:39:03.815865 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:03.815871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:03.816087 (+ 216us) server_negotiation.cc:407] Connection header received
0926 19:39:03.816238 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:03.816242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:03.816303 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:03.816375 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:03.818010 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.818699 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.820462 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.821124 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.821275 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:03.825116 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:03.825122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:03.825127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:03.825185 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:03.827610 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.828185 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.828188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.828191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.828293 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.828619 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.828623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.828625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.828865 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:03.829041 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:03.829349 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 19:39:03.829516 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 19:39:03.831049 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 2818
I20260926 19:39:03.842260 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:03.831860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.2.164.65:37261)
0926 19:39:03.832002 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:03.832008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:03.832295 (+ 287us) server_negotiation.cc:407] Connection header received
0926 19:39:03.832403 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:03.832406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:03.832456 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:03.832537 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:03.833640 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.834269 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.835431 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.835821 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.835919 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:03.837891 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:03.837895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:03.837898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:03.837924 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:03.840030 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.840579 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.840584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.840587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.840699 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.841011 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.841015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.841019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.841288 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:03.841486 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:03.841763 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 19:39:03.841953 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:39:03.842829 2941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:03.922039 2946 heartbeater.cc:503] Master 127.2.164.126:35101 was elected leader, sending a full tablet report...
I20260926 19:39:03.922618 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 2818
W20260926 19:39:03.934661 2742 connection.cc:535] server connection from 127.2.164.65:42641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 19:39:03.935034 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 2952
Sep 26 19:39:03 dist-test-slave-mrw4 krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:39:03.949858 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:03.935778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.2.164.66:45753)
0926 19:39:03.936325 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:03.936333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:03.936358 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:39:03.936416 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:03.936422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:03.936632 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:03.936758 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:03.938036 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.939026 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.940285 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:03.940844 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:03.941025 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:03.944880 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:03.944891 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:03.944905 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:03.944937 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:03.947504 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.948106 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.948112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.948115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.948226 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:03.948576 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:03.948579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:03.948581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:03.948881 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:03.949102 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:03.949390 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 19:39:03.949585 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":313,"threads_started":1}
I20260926 19:39:03.951054 3075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:04.029724 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 2952
I20260926 19:39:04.041816 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 3085
Sep 26 19:39:04 dist-test-slave-mrw4 krb5kdc[2712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:39:04.056607 3229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:04.042735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.2.164.67:39185)
0926 19:39:04.043316 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:04.043323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:04.043350 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:39:04.043406 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:04.043412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:04.043628 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:04.043744 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:04.044857 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:04.045906 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:04.047089 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:04.047619 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:04.047709 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:04.051536 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:04.051547 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:04.051560 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:04.051594 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:04.054223 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:04.054809 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:04.054816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:04.054818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:04.054963 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:04.055336 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:04.055340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:04.055343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:04.055668 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:04.055893 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:04.056149 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 19:39:04.056330 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":306,"threads_started":1}
I20260926 19:39:04.057981 3208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:04.137867 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 3085
I20260926 19:39:04.149958 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 2725
I20260926 19:39:04.151082 2787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:04.250555 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 2725
2026-09-26T19:39:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3530 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:04 dist-test-slave-mrw4 krb5kdc[3234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:04 dist-test-slave-mrw4 krb5kdc[3234](info): set up 2 sockets
Sep 26 19:39:04 dist-test-slave-mrw4 krb5kdc[3234](info): commencing operation
krb5kdc: starting...
W20260926 19:39:05.228834 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.004s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:05 dist-test-slave-mrw4 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:39:05.388746 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:42069
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:42069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:05.683782 3247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:05.684039 3247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:05.684166 3247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:05.690994 3247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:05.691095 3247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:05.691175 3247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:05.691248 3247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:05.691310 3247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:05.700834 3247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:42069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:42069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:05.702243 3247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:05.703548 3247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:05.714151 3252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:05.714192 3255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:05.714236 3253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:05.715173 3247 server_base.cc:1034] running on GCE node
I20260926 19:39:05.715828 3247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:05.717059 3247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:05.718261 3247 hybrid_clock.cc:648] HybridClock initialized: now 1790451545718202 us; error 41 us; skew 500 ppm
I20260926 19:39:05.718564 3247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:05 dist-test-slave-mrw4 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:05.721410 3247 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:39:05.722733 3247 webserver.cc:460] Webserver started at http://127.2.164.126:44067/ using document root <none> and password file <none>
I20260926 19:39:05.723362 3247 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:05.723469 3247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:05.723734 3247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:05.726508 3247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "d4703154443d46e78fccc732c38bdf14"
format_stamp: "Formatted at 2026-09-26 19:39:05 on dist-test-slave-mrw4"
server_key: "5fba6ff6299431aff5cd6918f47474c8"
server_key_iv: "59249a60d98b6f4fdad0f4d9d290a8ab"
server_key_version: "encryptionkey@0"
I20260926 19:39:05.727174 3247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "d4703154443d46e78fccc732c38bdf14"
format_stamp: "Formatted at 2026-09-26 19:39:05 on dist-test-slave-mrw4"
server_key: "5fba6ff6299431aff5cd6918f47474c8"
server_key_iv: "59249a60d98b6f4fdad0f4d9d290a8ab"
server_key_version: "encryptionkey@0"
I20260926 19:39:05.731837 3247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:39:05.735087 3262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:05.736426 3247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:39:05.736606 3247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "d4703154443d46e78fccc732c38bdf14"
format_stamp: "Formatted at 2026-09-26 19:39:05 on dist-test-slave-mrw4"
server_key: "5fba6ff6299431aff5cd6918f47474c8"
server_key_iv: "59249a60d98b6f4fdad0f4d9d290a8ab"
server_key_version: "encryptionkey@0"
I20260926 19:39:05.736783 3247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:05.754938 3247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:05.758854 3247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:05.759092 3247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:05.775144 3247 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:42069
I20260926 19:39:05.775163 3313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:42069 every 8 connection(s)
I20260926 19:39:05.776436 3247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:05.779944 3314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:05.781462 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 3247
I20260926 19:39:05.781685 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:05.782454 2705 external_mini_cluster.cc:1428] Setting key 759045dc03be1b85dfe74332de5e5ee2
I20260926 19:39:05.792079 3314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14: Bootstrap starting.
I20260926 19:39:05.794914 3314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14: Neither blocks nor log segments found. Creating new log.
Sep 26 19:39:05 dist-test-slave-mrw4 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:05.795835 3314 log.cc:824] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:05.798470 3314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14: No bootstrap required, opened a new log
I20260926 19:39:05.802687 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:05.784439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49182 (local address 127.2.164.126:42069)
0926 19:39:05.785212 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:05.785227 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:05.785268 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:39:05.786834 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:05.786857 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:05.787234 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:05.787560 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:05.788885 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:05.790214 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:05.791382 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:05.792001 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:05.792170 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:05.795985 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:05.795993 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:05.796006 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:05.796044 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:05.799008 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:05.799603 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:05.799609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:05.799614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:05.799787 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:05.800122 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:05.800125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:05.800127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:05.800631 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:05.800912 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:05.801413 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 19:39:05.801761 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":304,"threads_started":1}
I20260926 19:39:05.805794 3314 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4703154443d46e78fccc732c38bdf14" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42069 } }
I20260926 19:39:05.806119 3314 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:05.806207 3314 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4703154443d46e78fccc732c38bdf14, State: Initialized, Role: FOLLOWER
I20260926 19:39:05.806689 3314 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4703154443d46e78fccc732c38bdf14" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42069 } }
I20260926 19:39:05.806881 3314 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:05.807008 3314 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:05.807142 3314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:05.809053 3314 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4703154443d46e78fccc732c38bdf14" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42069 } }
I20260926 19:39:05.809512 3314 leader_election.cc:302] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4703154443d46e78fccc732c38bdf14; no voters:
I20260926 19:39:05.810024 3314 leader_election.cc:288] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:05.810137 3319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:05.811378 3319 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [term 1 LEADER]: Becoming Leader. State: Replica: d4703154443d46e78fccc732c38bdf14, State: Running, Role: LEADER
I20260926 19:39:05.811833 3319 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4703154443d46e78fccc732c38bdf14" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42069 } }
I20260926 19:39:05.812043 3314 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:05.814399 3320 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4703154443d46e78fccc732c38bdf14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4703154443d46e78fccc732c38bdf14" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42069 } } }
I20260926 19:39:05.814460 3321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4703154443d46e78fccc732c38bdf14. Latest consensus state: current_term: 1 leader_uuid: "d4703154443d46e78fccc732c38bdf14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4703154443d46e78fccc732c38bdf14" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42069 } } }
I20260926 19:39:05.814738 3320 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:05.814788 3321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:05.820930 3325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:05.824791 3325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:05.831653 3325 catalog_manager.cc:1347] Generated new cluster ID: bd169a7f099741bf8221c1db0b5e2250
I20260926 19:39:05.831748 3325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:05.843353 3325 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:05.845067 3325 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:05.858893 3325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4703154443d46e78fccc732c38bdf14: Generated new TSK 0
I20260926 19:39:05.859786 3325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:39:05.904111 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42069
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:06.212522 3340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:06.212815 3340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:06.212950 3340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:06.220153 3340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:06.220259 3340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:06.220376 3340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:06.229624 3340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42069
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:06.230713 3340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:06.231966 3340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:06.244033 3345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:06.244007 3346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:06.244464 3348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:06.245049 3340 server_base.cc:1034] running on GCE node
I20260926 19:39:06.245592 3340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:06.246588 3340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:06.247825 3340 hybrid_clock.cc:648] HybridClock initialized: now 1790451546247764 us; error 47 us; skew 500 ppm
I20260926 19:39:06.248134 3340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:06 dist-test-slave-mrw4 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:06.254277 3340 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:39:06.255601 3340 webserver.cc:460] Webserver started at http://127.2.164.65:42779/ using document root <none> and password file <none>
I20260926 19:39:06.256218 3340 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:06.256325 3340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:06.256587 3340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:06.259294 3340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "6852ae5a22474557833e6a042ede2cf4"
format_stamp: "Formatted at 2026-09-26 19:39:06 on dist-test-slave-mrw4"
server_key: "5cae5fa80386f8b96e78ccad501f1f1c"
server_key_iv: "26c6b24aadd9fa35110a764f1286df10"
server_key_version: "encryptionkey@0"
I20260926 19:39:06.260011 3340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "6852ae5a22474557833e6a042ede2cf4"
format_stamp: "Formatted at 2026-09-26 19:39:06 on dist-test-slave-mrw4"
server_key: "5cae5fa80386f8b96e78ccad501f1f1c"
server_key_iv: "26c6b24aadd9fa35110a764f1286df10"
server_key_version: "encryptionkey@0"
I20260926 19:39:06.264760 3340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:39:06.267885 3355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:06.269078 3340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 19:39:06.269237 3340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "6852ae5a22474557833e6a042ede2cf4"
format_stamp: "Formatted at 2026-09-26 19:39:06 on dist-test-slave-mrw4"
server_key: "5cae5fa80386f8b96e78ccad501f1f1c"
server_key_iv: "26c6b24aadd9fa35110a764f1286df10"
server_key_version: "encryptionkey@0"
I20260926 19:39:06.269393 3340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:06.288810 3340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:06.292292 3340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:06.292526 3340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:06.293421 3340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:06.294615 3340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:06.294689 3340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:06.294803 3340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:06.294854 3340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:06.322505 3340 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:35621
I20260926 19:39:06.322561 3467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:35621 every 8 connection(s)
I20260926 19:39:06.323626 3340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:06.332535 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 3340
I20260926 19:39:06.332736 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:06.333494 2705 external_mini_cluster.cc:1428] Setting key 7684758229acd2934452e6877a353536
Sep 26 19:39:06 dist-test-slave-mrw4 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:06.342648 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:06.325899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:36087 (local address 127.2.164.126:42069)
0926 19:39:06.326321 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:06.326327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:06.327287 (+ 960us) server_negotiation.cc:407] Connection header received
0926 19:39:06.328958 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:06.328962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:06.329025 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:06.329116 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:06.330514 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:06.331204 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:06.332630 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:06.333246 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:06.333371 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:06.337001 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:06.337007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:06.337012 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:06.337048 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:06.339810 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:06.340620 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:06.340623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:06.340625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:06.340738 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:06.341187 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:06.341192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:06.341196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:06.341467 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:06.341568 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:06.342260 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 19:39:06.342402 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":204,"threads_started":1}
I20260926 19:39:06.343920 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:06.326458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42069 (local address 127.2.164.65:36087)
0926 19:39:06.327128 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:06.327171 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:06.328717 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:06.329340 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:06.329348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:06.329839 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:06.330373 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:06.330390 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:06.331325 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:06.331328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:06.332494 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:06.332505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:06.333435 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:06.333440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:06.333637 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:06.334482 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:06.334515 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:06.336802 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:06.339940 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:06.339949 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:06.339964 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:06.340528 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:06.340852 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:06.340856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:06.340859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:06.341064 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:06.341574 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:06.341582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:06.341943 (+ 361us) client_negotiation.cc:769] Sending connection context
0926 19:39:06.342223 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 19:39:06.342685 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":259,"threads_started":1}
I20260926 19:39:06.345323 3468 heartbeater.cc:347] Connected to a master server at 127.2.164.126:42069
I20260926 19:39:06.345589 3468 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:06.346256 3468 heartbeater.cc:511] Master 127.2.164.126:42069 requested a full tablet report, sending...
I20260926 19:39:06.348191 3279 ts_manager.cc:194] Registered new tserver with Master: 6852ae5a22474557833e6a042ede2cf4 (127.2.164.65:35621)
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260926 19:39:06.349630 3279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:36087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:39:06.369496 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42069
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:06.664113 3474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:06.664366 3474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:06.664482 3474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:06.671255 3474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:06.671357 3474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:06.671499 3474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:06.680435 3474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42069
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:06.681555 3474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:06.682749 3474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:06.694260 3479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:06.694247 3480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:06.694294 3482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:06.695250 3474 server_base.cc:1034] running on GCE node
I20260926 19:39:06.695822 3474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:06.696703 3474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:06.697921 3474 hybrid_clock.cc:648] HybridClock initialized: now 1790451546697869 us; error 39 us; skew 500 ppm
I20260926 19:39:06.698251 3474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:06 dist-test-slave-mrw4 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:06.701054 3474 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:39:06.702387 3474 webserver.cc:460] Webserver started at http://127.2.164.66:42931/ using document root <none> and password file <none>
I20260926 19:39:06.703117 3474 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:06.703248 3474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:06.703603 3474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:06.706775 3474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc"
format_stamp: "Formatted at 2026-09-26 19:39:06 on dist-test-slave-mrw4"
server_key: "7cbced6f13d2e849a0569359087c2f6b"
server_key_iv: "bab1e310001c749553f0c296b9bb7614"
server_key_version: "encryptionkey@0"
I20260926 19:39:06.707499 3474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc"
format_stamp: "Formatted at 2026-09-26 19:39:06 on dist-test-slave-mrw4"
server_key: "7cbced6f13d2e849a0569359087c2f6b"
server_key_iv: "bab1e310001c749553f0c296b9bb7614"
server_key_version: "encryptionkey@0"
I20260926 19:39:06.712301 3474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 19:39:06.715392 3489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:06.716579 3474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:06.716710 3474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc"
format_stamp: "Formatted at 2026-09-26 19:39:06 on dist-test-slave-mrw4"
server_key: "7cbced6f13d2e849a0569359087c2f6b"
server_key_iv: "bab1e310001c749553f0c296b9bb7614"
server_key_version: "encryptionkey@0"
I20260926 19:39:06.716866 3474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:06.736848 3474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:06.740404 3474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:06.740644 3474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:06.741590 3474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:06.742738 3474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:06.742805 3474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:06.742908 3474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:06.742961 3474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:06.769996 3474 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:43315
I20260926 19:39:06.770047 3601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:43315 every 8 connection(s)
I20260926 19:39:06.771128 3474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:06.771940 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 3474
I20260926 19:39:06.772119 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:06.772637 2705 external_mini_cluster.cc:1428] Setting key 5696c74539f8c2638a7cb97322560541
Sep 26 19:39:06 dist-test-slave-mrw4 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:06.791678 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:06.773756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:46471 (local address 127.2.164.126:42069)
0926 19:39:06.773930 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:06.773936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:06.775282 (+ 1346us) server_negotiation.cc:407] Connection header received
0926 19:39:06.777642 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:06.777647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:06.777751 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:06.777858 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:06.779622 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:06.780619 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:06.782840 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:06.783249 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:06.783361 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:06.786422 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:06.786426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:06.786429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:06.786458 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:06.788955 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:06.789632 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:06.789636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:06.789638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:06.789757 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:06.790123 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:06.790127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:06.790129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:06.790379 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:06.790505 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:06.791120 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 19:39:06.791371 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260926 19:39:06.792589 3604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:06.774271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42069 (local address 127.2.164.66:46471)
0926 19:39:06.775110 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:06.775163 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:06.777367 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:06.778096 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:06.778109 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:06.778761 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:06.779443 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:06.779470 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:06.780788 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:06.780793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:06.782698 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:06.782713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:06.783456 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:06.783463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:06.783680 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:06.784258 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:06.784280 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:06.786232 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:06.789070 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:06.789086 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:06.789101 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:06.789513 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:06.789849 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:06.789854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:06.789858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:06.790018 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:06.790491 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:06.790498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:06.790786 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 19:39:06.790996 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 19:39:06.791440 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":289,"threads_started":1}
I20260926 19:39:06.793844 3602 heartbeater.cc:347] Connected to a master server at 127.2.164.126:42069
I20260926 19:39:06.794128 3602 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
I20260926 19:39:06.794909 3602 heartbeater.cc:511] Master 127.2.164.126:42069 requested a full tablet report, sending...
I20260926 19:39:06.796211 3279 ts_manager.cc:194] Registered new tserver with Master: bd52ccf7d4bb4494b7b5ae54c09496fc (127.2.164.66:43315)
I20260926 19:39:06.796967 3279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:46471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:39:06.813548 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42069
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:07.106829 3607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:07.107089 3607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:07.107210 3607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:07.114156 3607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:07.114264 3607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:07.114416 3607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:07.123606 3607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42069
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:07.124806 3607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:07.126156 3607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:07.138116 3615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:07.138093 3612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:07.138175 3613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:07.139025 3607 server_base.cc:1034] running on GCE node
I20260926 19:39:07.139547 3607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:07.140477 3607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:07.141674 3607 hybrid_clock.cc:648] HybridClock initialized: now 1790451547141630 us; error 36 us; skew 500 ppm
I20260926 19:39:07.141939 3607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:07 dist-test-slave-mrw4 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451547, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:07.144863 3607 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:39:07.146157 3607 webserver.cc:460] Webserver started at http://127.2.164.67:40413/ using document root <none> and password file <none>
I20260926 19:39:07.146761 3607 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:07.146853 3607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:07.147092 3607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:07.149765 3607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "ab5f816840e24ae8bbf337a106fadbd0"
format_stamp: "Formatted at 2026-09-26 19:39:07 on dist-test-slave-mrw4"
server_key: "4596e50de868c97076b9984149dc4cbb"
server_key_iv: "6df5f460021eb1fb5e8df54b4e52353d"
server_key_version: "encryptionkey@0"
I20260926 19:39:07.150440 3607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "ab5f816840e24ae8bbf337a106fadbd0"
format_stamp: "Formatted at 2026-09-26 19:39:07 on dist-test-slave-mrw4"
server_key: "4596e50de868c97076b9984149dc4cbb"
server_key_iv: "6df5f460021eb1fb5e8df54b4e52353d"
server_key_version: "encryptionkey@0"
I20260926 19:39:07.155218 3607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:39:07.158288 3622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:07.159475 3607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:39:07.159605 3607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "ab5f816840e24ae8bbf337a106fadbd0"
format_stamp: "Formatted at 2026-09-26 19:39:07 on dist-test-slave-mrw4"
server_key: "4596e50de868c97076b9984149dc4cbb"
server_key_iv: "6df5f460021eb1fb5e8df54b4e52353d"
server_key_version: "encryptionkey@0"
I20260926 19:39:07.159757 3607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:07.175661 3607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:07.179433 3607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:07.179658 3607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:07.180470 3607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:07.181687 3607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:07.181756 3607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:07.181841 3607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:07.181882 3607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:07.210198 3607 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:43787
I20260926 19:39:07.210222 3734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:43787 every 8 connection(s)
I20260926 19:39:07.211393 3607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:07.215975 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 3607
I20260926 19:39:07.216136 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:07.216529 2705 external_mini_cluster.cc:1428] Setting key 6fbccf27c242e35a5c93b26b63f66691
Sep 26 19:39:07 dist-test-slave-mrw4 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451547, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:07.229929 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.213877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:41251 (local address 127.2.164.126:42069)
0926 19:39:07.214065 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.214071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.215300 (+ 1229us) server_negotiation.cc:407] Connection header received
0926 19:39:07.217122 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.217126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.217263 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.217374 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:07.219235 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.220048 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.221656 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.222049 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.222139 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.225089 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:07.225094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:07.225097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:07.225128 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:07.227450 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.228034 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.228038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.228040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.228140 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.228450 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.228453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.228455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.228690 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:07.228796 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.229417 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.229607 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:39:07.230949 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.214400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42069 (local address 127.2.164.67:41251)
0926 19:39:07.215142 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:07.215179 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:07.216851 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:07.217617 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:07.217627 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:07.218314 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:07.219053 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.219077 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.220174 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.220178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.221548 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.221559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.222208 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.222213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.222400 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.222948 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:07.222970 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:07.224907 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:07.227560 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:07.227566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:07.227577 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:07.227952 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:07.228208 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:07.228211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:07.228213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:07.228376 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:07.228784 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:07.228789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:07.229064 (+ 275us) client_negotiation.cc:769] Sending connection context
0926 19:39:07.229313 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 19:39:07.229691 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":286,"threads_started":1}
I20260926 19:39:07.232209 3735 heartbeater.cc:347] Connected to a master server at 127.2.164.126:42069
I20260926 19:39:07.232489 3735 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:07.233150 3735 heartbeater.cc:511] Master 127.2.164.126:42069 requested a full tablet report, sending...
I20260926 19:39:07.234529 3279 ts_manager.cc:194] Registered new tserver with Master: ab5f816840e24ae8bbf337a106fadbd0 (127.2.164.67:43787)
I20260926 19:39:07.235275 3279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:41251
I20260926 19:39:07.243495 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:39:07.260763 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.249151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.2.164.126:42069)
0926 19:39:07.249301 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.249305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.249750 (+ 445us) server_negotiation.cc:407] Connection header received
0926 19:39:07.249859 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.249863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.249920 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.249999 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:07.251263 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.251955 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.253204 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.253609 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.253708 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.255684 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:07.255688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:07.255691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:07.255719 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:07.258380 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.258966 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.258972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.258976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.259107 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.259496 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.259499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.259501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.259786 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:07.259952 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.260336 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.260518 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:39:07.266906 3279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:39:07.273459 3279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:07.295060 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.284874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.2.164.66:43315)
0926 19:39:07.287340 (+ 2466us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.287349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.287381 (+ 32us) server_negotiation.cc:407] Connection header received
0926 19:39:07.287473 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.287480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.287745 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.288574 (+ 829us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:07.288843 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.290289 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.292906 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.294059 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.294190 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.294385 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.294616 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.294761 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2316,"thread_start_us":2198,"threads_started":1}
I20260926 19:39:07.295159 3747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.284740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:43315 (local address 127.0.0.1:53544)
0926 19:39:07.286528 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:07.286622 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:07.286819 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:07.287976 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:07.287984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:07.288017 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:07.288387 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.288398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.290435 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.290441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.292754 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.292776 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.294239 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.294246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.294421 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.294460 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 19:39:07.294564 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 19:39:07.294748 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1531,"mutex_wait_us":26,"thread_start_us":3142,"threads_started":1}
I20260926 19:39:07.296082 3751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.284768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43787 (local address 127.0.0.1:47216)
0926 19:39:07.287595 (+ 2827us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:07.287611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:07.287706 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:07.289334 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:07.289338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:07.289365 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:07.289670 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.289679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.291340 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.291345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.293314 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.293327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.294963 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.294973 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.295493 (+ 520us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.295528 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:39:07.295588 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 19:39:07.295703 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2728,"mutex_wait_us":5,"thread_start_us":1354,"threads_started":1}
I20260926 19:39:07.296691 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.287378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.2.164.65:35621)
0926 19:39:07.288389 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.288398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.288425 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:39:07.288511 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.288518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.288750 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.289549 (+ 799us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:07.289689 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.290942 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.293192 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.294548 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.294676 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.294862 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.296290 (+ 1428us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.296442 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":1798,"threads_started":1}
I20260926 19:39:07.296764 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.285354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.2.164.67:43787)
0926 19:39:07.288794 (+ 3440us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.288801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.288829 (+ 28us) server_negotiation.cc:407] Connection header received
0926 19:39:07.288912 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.288918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.289103 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.289858 (+ 755us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:07.290081 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.291202 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.293475 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.294746 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.295918 (+ 1172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.296126 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.296317 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.296462 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3273,"thread_start_us":3638,"threads_started":1}
I20260926 19:39:07.295825 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.286984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35621 (local address 127.0.0.1:42532)
0926 19:39:07.288171 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:07.288196 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:07.288298 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:07.288951 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:07.288955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:07.288980 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:07.289388 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.289400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.291070 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.291076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.293028 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.293041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.294989 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.294995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.295199 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.295227 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 19:39:07.295303 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 19:39:07.295420 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"mutex_wait_us":59,"thread_start_us":395,"threads_started":1}
I20260926 19:39:07.298959 3537 tablet_service.cc:1467] Processing CreateTablet for tablet 9db04eb462994d128b859c916f10ad5e (DEFAULT_TABLE table=test-table [id=197018ed7f464fa58b4bad92fb591741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:07.299667 3403 tablet_service.cc:1467] Processing CreateTablet for tablet 9db04eb462994d128b859c916f10ad5e (DEFAULT_TABLE table=test-table [id=197018ed7f464fa58b4bad92fb591741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:07.300365 3537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db04eb462994d128b859c916f10ad5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:07.300308 3670 tablet_service.cc:1467] Processing CreateTablet for tablet 9db04eb462994d128b859c916f10ad5e (DEFAULT_TABLE table=test-table [id=197018ed7f464fa58b4bad92fb591741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:07.300782 3403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db04eb462994d128b859c916f10ad5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:07.301594 3670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db04eb462994d128b859c916f10ad5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:07.310772 3753 tablet_bootstrap.cc:492] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4: Bootstrap starting.
I20260926 19:39:07.313853 3754 tablet_bootstrap.cc:492] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc: Bootstrap starting.
I20260926 19:39:07.314738 3753 tablet_bootstrap.cc:654] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:07.315914 3753 log.cc:824] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:07.317031 3754 tablet_bootstrap.cc:654] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:07.317227 3755 tablet_bootstrap.cc:492] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0: Bootstrap starting.
I20260926 19:39:07.318217 3754 log.cc:824] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:07.318779 3753 tablet_bootstrap.cc:492] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4: No bootstrap required, opened a new log
I20260926 19:39:07.319084 3753 ts_tablet_manager.cc:1397] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 19:39:07.320508 3755 tablet_bootstrap.cc:654] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:07.321049 3754 tablet_bootstrap.cc:492] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc: No bootstrap required, opened a new log
I20260926 19:39:07.321326 3754 ts_tablet_manager.cc:1397] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 19:39:07.321606 3755 log.cc:824] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:07.326902 3755 tablet_bootstrap.cc:492] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0: No bootstrap required, opened a new log
I20260926 19:39:07.327140 3755 ts_tablet_manager.cc:1397] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 19:39:07.327697 3753 raft_consensus.cc:348] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 43315 } } peers { permanent_uuid: "ab5f816840e24ae8bbf337a106fadbd0" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43787 } } peers { permanent_uuid: "6852ae5a22474557833e6a042ede2cf4" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35621 } }
I20260926 19:39:07.328128 3753 raft_consensus.cc:374] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:07.328243 3753 raft_consensus.cc:729] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6852ae5a22474557833e6a042ede2cf4, State: Initialized, Role: FOLLOWER
I20260926 19:39:07.328917 3753 consensus_queue.cc:260] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 43315 } } peers { permanent_uuid: "ab5f816840e24ae8bbf337a106fadbd0" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43787 } } peers { permanent_uuid: "6852ae5a22474557833e6a042ede2cf4" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35621 } }
I20260926 19:39:07.330946 3754 raft_consensus.cc:348] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 43315 } } peers { permanent_uuid: "ab5f816840e24ae8bbf337a106fadbd0" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43787 } } peers { permanent_uuid: "6852ae5a22474557833e6a042ede2cf4" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35621 } }
I20260926 19:39:07.331369 3754 raft_consensus.cc:374] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:07.331483 3754 raft_consensus.cc:729] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd52ccf7d4bb4494b7b5ae54c09496fc, State: Initialized, Role: FOLLOWER
I20260926 19:39:07.332120 3754 consensus_queue.cc:260] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 43315 } } peers { permanent_uuid: "ab5f816840e24ae8bbf337a106fadbd0" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43787 } } peers { permanent_uuid: "6852ae5a22474557833e6a042ede2cf4" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35621 } }
I20260926 19:39:07.338635 3755 raft_consensus.cc:348] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 43315 } } peers { permanent_uuid: "ab5f816840e24ae8bbf337a106fadbd0" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43787 } } peers { permanent_uuid: "6852ae5a22474557833e6a042ede2cf4" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35621 } }
I20260926 19:39:07.339107 3755 raft_consensus.cc:374] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:07.339231 3755 raft_consensus.cc:729] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5f816840e24ae8bbf337a106fadbd0, State: Initialized, Role: FOLLOWER
I20260926 19:39:07.340459 3755 consensus_queue.cc:260] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 43315 } } peers { permanent_uuid: "ab5f816840e24ae8bbf337a106fadbd0" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43787 } } peers { permanent_uuid: "6852ae5a22474557833e6a042ede2cf4" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35621 } }
I20260926 19:39:07.340939 3753 ts_tablet_manager.cc:1428] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4: Time spent starting tablet: real 0.022s user 0.016s sys 0.006s
I20260926 19:39:07.340970 3468 heartbeater.cc:503] Master 127.2.164.126:42069 was elected leader, sending a full tablet report...
I20260926 19:39:07.342289 3754 ts_tablet_manager.cc:1428] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc: Time spent starting tablet: real 0.021s user 0.011s sys 0.009s
I20260926 19:39:07.342973 3602 heartbeater.cc:503] Master 127.2.164.126:42069 was elected leader, sending a full tablet report...
I20260926 19:39:07.343488 3735 heartbeater.cc:503] Master 127.2.164.126:42069 was elected leader, sending a full tablet report...
I20260926 19:39:07.344012 3755 ts_tablet_manager.cc:1428] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0: Time spent starting tablet: real 0.017s user 0.009s sys 0.004s
I20260926 19:39:07.357944 3759 raft_consensus.cc:482] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:07.358143 3759 raft_consensus.cc:504] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 43315 } } peers { permanent_uuid: "ab5f816840e24ae8bbf337a106fadbd0" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43787 } } peers { permanent_uuid: "6852ae5a22474557833e6a042ede2cf4" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35621 } }
I20260926 19:39:07.359457 3759 leader_election.cc:288] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd52ccf7d4bb4494b7b5ae54c09496fc (127.2.164.66:43315), ab5f816840e24ae8bbf337a106fadbd0 (127.2.164.67:43787)
I20260926 19:39:07.366492 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.359863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:37199 (local address 127.2.164.67:43787)
0926 19:39:07.360092 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.360098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.360619 (+ 521us) server_negotiation.cc:407] Connection header received
0926 19:39:07.360734 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.360739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.360815 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.360897 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:07.361620 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.362679 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.364730 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.365686 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.365843 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.365913 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.366197 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.366316 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 19:39:07.366705 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.359735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43787 (local address 127.2.164.65:37199)
0926 19:39:07.360532 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:07.360550 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:07.360634 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:07.360915 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:07.360920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:07.360995 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:07.361475 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.361490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.362871 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.362876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.364591 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.364603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.365876 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.365883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.366034 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.366057 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:39:07.366156 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 19:39:07.366284 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":93,"spinlock_wait_cycles":16768,"thread_start_us":485,"threads_started":1}
I20260926 19:39:07.366973 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.359869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:39905 (local address 127.2.164.66:43315)
0926 19:39:07.360040 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.360046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.360564 (+ 518us) server_negotiation.cc:407] Connection header received
0926 19:39:07.360735 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.360739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.360808 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.360898 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:07.361636 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.362679 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.365320 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.366047 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.366197 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.366308 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.366665 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.366794 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:39:07.367566 3762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.359654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:43315 (local address 127.2.164.65:39905)
0926 19:39:07.360464 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:07.360482 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:07.360623 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:07.360915 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:07.360920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:07.360944 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:07.361514 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.361527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.362805 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.362810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.364593 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:07.364604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.366288 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.366293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:07.366484 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.366514 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 19:39:07.367116 (+ 602us) client_negotiation.cc:240] Negotiation successful
0926 19:39:07.367242 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":53,"thread_start_us":375,"threads_started":1}
I20260926 19:39:07.367679 3690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db04eb462994d128b859c916f10ad5e" candidate_uuid: "6852ae5a22474557833e6a042ede2cf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5f816840e24ae8bbf337a106fadbd0" is_pre_election: true
I20260926 19:39:07.368114 3690 raft_consensus.cc:2454] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6852ae5a22474557833e6a042ede2cf4 in term 0.
I20260926 19:39:07.368399 3557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db04eb462994d128b859c916f10ad5e" candidate_uuid: "6852ae5a22474557833e6a042ede2cf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc" is_pre_election: true
I20260926 19:39:07.368688 3356 leader_election.cc:302] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6852ae5a22474557833e6a042ede2cf4, ab5f816840e24ae8bbf337a106fadbd0; no voters:
I20260926 19:39:07.368842 3557 raft_consensus.cc:2454] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6852ae5a22474557833e6a042ede2cf4 in term 0.
I20260926 19:39:07.369060 3759 raft_consensus.cc:2781] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:07.369230 3759 raft_consensus.cc:482] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:07.369339 3759 raft_consensus.cc:3037] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:07.371392 3759 raft_consensus.cc:504] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 43315 } } peers { permanent_uuid: "ab5f816840e24ae8bbf337a106fadbd0" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43787 } } peers { permanent_uuid: "6852ae5a22474557833e6a042ede2cf4" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35621 } }
I20260926 19:39:07.371925 3759 leader_election.cc:288] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [CANDIDATE]: Term 1 election: Requested vote from peers bd52ccf7d4bb4494b7b5ae54c09496fc (127.2.164.66:43315), ab5f816840e24ae8bbf337a106fadbd0 (127.2.164.67:43787)
I20260926 19:39:07.372241 3557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db04eb462994d128b859c916f10ad5e" candidate_uuid: "6852ae5a22474557833e6a042ede2cf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc"
I20260926 19:39:07.372332 3690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db04eb462994d128b859c916f10ad5e" candidate_uuid: "6852ae5a22474557833e6a042ede2cf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5f816840e24ae8bbf337a106fadbd0"
I20260926 19:39:07.372419 3557 raft_consensus.cc:3037] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:07.372498 3690 raft_consensus.cc:3037] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:07.375411 3690 raft_consensus.cc:2454] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6852ae5a22474557833e6a042ede2cf4 in term 1.
I20260926 19:39:07.375411 3557 raft_consensus.cc:2454] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6852ae5a22474557833e6a042ede2cf4 in term 1.
I20260926 19:39:07.375723 3356 leader_election.cc:302] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6852ae5a22474557833e6a042ede2cf4, ab5f816840e24ae8bbf337a106fadbd0; no voters:
I20260926 19:39:07.375969 3759 raft_consensus.cc:2781] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:07.376350 3759 raft_consensus.cc:686] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [term 1 LEADER]: Becoming Leader. State: Replica: 6852ae5a22474557833e6a042ede2cf4, State: Running, Role: LEADER
I20260926 19:39:07.376735 3759 consensus_queue.cc:237] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 43315 } } peers { permanent_uuid: "ab5f816840e24ae8bbf337a106fadbd0" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43787 } } peers { permanent_uuid: "6852ae5a22474557833e6a042ede2cf4" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35621 } }
I20260926 19:39:07.380589 3277 catalog_manager.cc:5495] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6852ae5a22474557833e6a042ede2cf4 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "6852ae5a22474557833e6a042ede2cf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd52ccf7d4bb4494b7b5ae54c09496fc" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 43315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5f816840e24ae8bbf337a106fadbd0" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6852ae5a22474557833e6a042ede2cf4" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35621 } health_report { overall_health: HEALTHY } } }
Sep 26 19:39:07 dist-test-slave-mrw4 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:07.422101 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.408519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.2.164.65:35621)
0926 19:39:07.408710 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.408717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.409087 (+ 370us) server_negotiation.cc:407] Connection header received
0926 19:39:07.409260 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.409264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.409339 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.409431 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:07.410886 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.411579 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.412807 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.413240 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.413352 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.417021 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:07.417035 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:07.417049 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:07.417080 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:07.419777 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.420365 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.420372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.420374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.420516 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.420804 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.420807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.420809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.421209 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:07.421426 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.421629 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.421807 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 19:39:07 dist-test-slave-mrw4 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451547, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 19:39:07 dist-test-slave-mrw4 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451547, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
W20260926 19:39:07.462916 3736 tablet.cc:2369] T 9db04eb462994d128b859c916f10ad5e P ab5f816840e24ae8bbf337a106fadbd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:07.464365 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.449654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.2.164.65:35621)
0926 19:39:07.449826 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.449834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.450301 (+ 467us) server_negotiation.cc:407] Connection header received
0926 19:39:07.450494 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.450500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.450580 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.450680 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:07.452289 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.453324 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.454968 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.455500 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.455578 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.459886 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:07.459890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:07.459894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:07.459926 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:07.462058 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.462575 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.462579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.462581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.462696 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.463168 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.463172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.463175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.463416 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:07.463608 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.463904 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.464073 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 19:39:07 dist-test-slave-mrw4 krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 19:39:07 dist-test-slave-mrw4 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:07.502208 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.489084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.2.164.65:35621)
0926 19:39:07.489283 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.489289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.489610 (+ 321us) server_negotiation.cc:407] Connection header received
0926 19:39:07.489770 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.489774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.489860 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.489943 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:07.491271 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.491981 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.493307 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.493706 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.493829 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.497379 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:07.497383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:07.497387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:07.497418 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:07.499703 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.500403 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.500407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.500409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.500544 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.500902 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.500906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.500908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.501153 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:07.501414 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.501687 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.501909 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 19:39:07.503839 3383 tablet_service.cc:2908] Rejecting scan request for tablet 9db04eb462994d128b859c916f10ad5e: Uninitialized: clean time has not yet been initialized
I20260926 19:39:07.506680 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 3340
I20260926 19:39:07.518373 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.507419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.2.164.65:35621)
0926 19:39:07.507583 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.507589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.507878 (+ 289us) server_negotiation.cc:407] Connection header received
0926 19:39:07.508002 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.508006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.508058 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.508124 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:07.509225 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.509926 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.511041 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.511473 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.511587 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.513705 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:07.513709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:07.513712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:07.513740 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:07.516132 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.516694 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.516697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.516699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.516797 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.517118 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.517121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.517123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.517407 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:07.517588 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.517917 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.518060 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:39:07.519001 3463 generic_service.cc:196] Checking for leaks (request via RPC)
W20260926 19:39:07.523029 3603 tablet.cc:2369] T 9db04eb462994d128b859c916f10ad5e P bd52ccf7d4bb4494b7b5ae54c09496fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:39:07.605588 3469 tablet.cc:2369] T 9db04eb462994d128b859c916f10ad5e P 6852ae5a22474557833e6a042ede2cf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:07.606294 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 3340
I20260926 19:39:07.619393 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 3474
Sep 26 19:39:07 dist-test-slave-mrw4 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:39:07.633100 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.620050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.2.164.66:43315)
0926 19:39:07.620210 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.620216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.620277 (+ 61us) server_negotiation.cc:407] Connection header received
0926 19:39:07.620469 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.620475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.620540 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.620618 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:07.621864 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.622585 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.623781 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.624183 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.624289 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.628117 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:07.628130 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:07.628145 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:07.628179 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:07.630629 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.631240 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.631248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.631250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.631367 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.631697 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.631701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.631703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.632047 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:07.632332 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.632589 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.632780 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:39:07.633829 3597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:07.718161 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 3474
I20260926 19:39:07.731482 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 3607
Sep 26 19:39:07 dist-test-slave-mrw4 krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:39:07.744716 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:07.732202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.2.164.67:43787)
0926 19:39:07.732354 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:07.732358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:07.732375 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:39:07.732432 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:07.732436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:07.732498 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:07.732585 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:07.733775 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.734498 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.735715 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:07.736123 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:07.736232 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:07.739988 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:07.740001 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:07.740017 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:07.740052 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:07.742607 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.743202 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.743209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.743212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.743336 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:07.743593 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:07.743596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:07.743598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:07.743917 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:07.744166 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:07.744335 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 19:39:07.744483 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 19:39:07.745309 3730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:07.829135 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 3607
I20260926 19:39:07.841778 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 3247
I20260926 19:39:07.842808 3309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:07.946923 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 3247
2026-09-26T19:39:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3696 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:08 dist-test-slave-mrw4 krb5kdc[3783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:08 dist-test-slave-mrw4 krb5kdc[3783](info): set up 2 sockets
Sep 26 19:39:08 dist-test-slave-mrw4 krb5kdc[3783](info): commencing operation
krb5kdc: starting...
W20260926 19:39:08.932796 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:09 dist-test-slave-mrw4 krb5kdc[3783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:39:09.094834 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34743
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:09.394354 3796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:09.394629 3796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:09.394742 3796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:09.401713 3796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:09.401803 3796 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 19:39:09.401862 3796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:09.401937 3796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:09.402001 3796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:09.402069 3796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:09.411484 3796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:34743
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:09.412657 3796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:09.413945 3796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:09.425683 3801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:09.425617 3802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:09.425653 3804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:09.426422 3796 server_base.cc:1034] running on GCE node
I20260926 19:39:09.427099 3796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:09.428319 3796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:09.429514 3796 hybrid_clock.cc:648] HybridClock initialized: now 1790451549429454 us; error 35 us; skew 500 ppm
I20260926 19:39:09.429806 3796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:09 dist-test-slave-mrw4 krb5kdc[3783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451549, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:09.432590 3796 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:39:09.433883 3796 webserver.cc:460] Webserver started at http://127.2.164.126:39825/ using document root <none> and password file <none>
I20260926 19:39:09.434514 3796 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:09.434602 3796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:09.434890 3796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:09.437701 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "705046327de74bea86ef62d92aa1f527"
format_stamp: "Formatted at 2026-09-26 19:39:09 on dist-test-slave-mrw4"
server_key: "9068ff2afe5209807c17018cf5fc7cff"
server_key_iv: "2a54a361425836649f8655d9d561f94d"
server_key_version: "encryptionkey@0"
I20260926 19:39:09.438433 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "705046327de74bea86ef62d92aa1f527"
format_stamp: "Formatted at 2026-09-26 19:39:09 on dist-test-slave-mrw4"
server_key: "9068ff2afe5209807c17018cf5fc7cff"
server_key_iv: "2a54a361425836649f8655d9d561f94d"
server_key_version: "encryptionkey@0"
I20260926 19:39:09.443146 3796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:39:09.446323 3811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:09.447535 3796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:39:09.447672 3796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "705046327de74bea86ef62d92aa1f527"
format_stamp: "Formatted at 2026-09-26 19:39:09 on dist-test-slave-mrw4"
server_key: "9068ff2afe5209807c17018cf5fc7cff"
server_key_iv: "2a54a361425836649f8655d9d561f94d"
server_key_version: "encryptionkey@0"
I20260926 19:39:09.447849 3796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:09.468531 3796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:09.472383 3796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:09.472575 3796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:09.490281 3796 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:34743
I20260926 19:39:09.490301 3872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:34743 every 8 connection(s)
I20260926 19:39:09.491469 3796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:09.494897 3873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:09.497949 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 3796
I20260926 19:39:09.498126 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:09.498723 2705 external_mini_cluster.cc:1428] Setting key ba42d500d47823aa563d2ba6dfd656d5
I20260926 19:39:09.507329 3873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527: Bootstrap starting.
I20260926 19:39:09.510730 3873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:09.511643 3873 log.cc:824] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:39:09 dist-test-slave-mrw4 krb5kdc[3783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:09.514343 3873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527: No bootstrap required, opened a new log
I20260926 19:39:09.520615 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:09.500574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.2.164.126:34743)
0926 19:39:09.501369 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:09.501383 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:09.501434 (+ 51us) server_negotiation.cc:407] Connection header received
0926 19:39:09.503254 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:09.503283 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:09.503778 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:09.504215 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:09.505516 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:09.507395 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:09.508619 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:09.509643 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:09.509813 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:09.513563 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:09.513575 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:09.513594 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:09.513634 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:09.516724 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:09.517461 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:09.517467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:09.517473 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:09.517635 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:09.517998 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:09.518001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:09.518004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:09.518518 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:09.518808 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:09.519240 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 19:39:09.519626 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":328,"threads_started":1}
I20260926 19:39:09.521756 3873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705046327de74bea86ef62d92aa1f527" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34743 } }
I20260926 19:39:09.522181 3873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:09.522292 3873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705046327de74bea86ef62d92aa1f527, State: Initialized, Role: FOLLOWER
I20260926 19:39:09.522909 3873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705046327de74bea86ef62d92aa1f527" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34743 } }
I20260926 19:39:09.523180 3873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:09.523322 3873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:09.523499 3873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:09.525779 3873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705046327de74bea86ef62d92aa1f527" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34743 } }
I20260926 19:39:09.526197 3873 leader_election.cc:302] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 705046327de74bea86ef62d92aa1f527; no voters:
I20260926 19:39:09.526679 3873 leader_election.cc:288] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:09.526804 3878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:09.527410 3878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [term 1 LEADER]: Becoming Leader. State: Replica: 705046327de74bea86ef62d92aa1f527, State: Running, Role: LEADER
I20260926 19:39:09.527938 3878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705046327de74bea86ef62d92aa1f527" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34743 } }
I20260926 19:39:09.528517 3873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:09.530230 3880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 705046327de74bea86ef62d92aa1f527. Latest consensus state: current_term: 1 leader_uuid: "705046327de74bea86ef62d92aa1f527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705046327de74bea86ef62d92aa1f527" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34743 } } }
I20260926 19:39:09.530596 3880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:09.532047 3883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:09.532068 3879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "705046327de74bea86ef62d92aa1f527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705046327de74bea86ef62d92aa1f527" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34743 } } }
I20260926 19:39:09.532451 3879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:09.536433 3883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:09.544198 3883 catalog_manager.cc:1347] Generated new cluster ID: d0ac44122e4f4227a5f20b47b0ec933e
I20260926 19:39:09.544292 3883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:09.557677 3883 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:09.559373 3883 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:09.568182 3883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527: Generated new TSK 0
I20260926 19:39:09.569073 3883 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:39:09.616787 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34743
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:09.911998 3899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:09.912256 3899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:09.912403 3899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:09.919248 3899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:09.919364 3899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:09.919484 3899 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:39:09.919590 3899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:09.928817 3899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:34743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:09.929994 3899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:09.931263 3899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:09.942866 3904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:09.942898 3905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:09.943014 3907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:09.943864 3899 server_base.cc:1034] running on GCE node
I20260926 19:39:09.944448 3899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:09.945294 3899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:09.946516 3899 hybrid_clock.cc:648] HybridClock initialized: now 1790451549946450 us; error 48 us; skew 500 ppm
I20260926 19:39:09.946795 3899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:09 dist-test-slave-mrw4 krb5kdc[3783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451549, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:09.949854 3899 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:39:09.951164 3899 webserver.cc:460] Webserver started at http://127.2.164.65:33763/ using document root <none> and password file <none>
I20260926 19:39:09.951839 3899 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:09.951932 3899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:09.952217 3899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:09.955121 3899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "f04c989a08d2423aae06476aea8cc63a"
format_stamp: "Formatted at 2026-09-26 19:39:09 on dist-test-slave-mrw4"
server_key: "6f5d607c58f1f0b1626b81e3da6c42c1"
server_key_iv: "9571dcf81013be8639b75404327c234c"
server_key_version: "encryptionkey@0"
I20260926 19:39:09.955875 3899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "f04c989a08d2423aae06476aea8cc63a"
format_stamp: "Formatted at 2026-09-26 19:39:09 on dist-test-slave-mrw4"
server_key: "6f5d607c58f1f0b1626b81e3da6c42c1"
server_key_iv: "9571dcf81013be8639b75404327c234c"
server_key_version: "encryptionkey@0"
I20260926 19:39:09.960457 3899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260926 19:39:09.963577 3914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:09.964737 3899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:09.964869 3899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "f04c989a08d2423aae06476aea8cc63a"
format_stamp: "Formatted at 2026-09-26 19:39:09 on dist-test-slave-mrw4"
server_key: "6f5d607c58f1f0b1626b81e3da6c42c1"
server_key_iv: "9571dcf81013be8639b75404327c234c"
server_key_version: "encryptionkey@0"
I20260926 19:39:09.965046 3899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:09.993280 3899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:09.997360 3899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:09.997599 3899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:10.000104 3899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:10.000181 3899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:10.000310 3899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:10.000363 3899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:39:10 dist-test-slave-mrw4 krb5kdc[3783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451549, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:10.022131 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.000762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:38771 (local address 127.2.164.126:34743)
0926 19:39:10.000940 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:10.000947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:10.002400 (+ 1453us) server_negotiation.cc:407] Connection header received
0926 19:39:10.004650 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:10.004654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:10.004721 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:10.004786 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:10.007156 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.008091 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.010441 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.011006 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.011138 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.015573 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:10.015579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:10.015583 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:10.015618 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:10.018633 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.019553 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.019560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.019564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.019716 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.020271 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.020276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.020279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.020663 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:10.020805 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:10.021594 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 19:39:10.021806 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:39:10.023132 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.001362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34743 (local address 127.2.164.65:38771)
0926 19:39:10.002207 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:10.002261 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:10.004289 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:10.005369 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:10.005380 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:10.006154 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:10.006975 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.007008 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.008257 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.008265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.010297 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.010314 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.011249 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.011259 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.011571 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.012359 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:10.012389 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:10.015351 (+ 2962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:10.018766 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.018776 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.018791 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.019393 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.019839 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.019844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.019848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.020119 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.020804 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:10.020812 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:10.021224 (+ 412us) client_negotiation.cc:769] Sending connection context
0926 19:39:10.021487 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 19:39:10.021997 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":338,"threads_started":1}
I20260926 19:39:10.041414 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.029040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:33159 (local address 127.2.164.126:34743)
0926 19:39:10.029211 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:10.029217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:10.029811 (+ 594us) server_negotiation.cc:407] Connection header received
0926 19:39:10.030007 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:10.030013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:10.030082 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:10.030159 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:10.031030 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.031726 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.033396 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.034033 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.034163 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.036213 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:10.036217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:10.036220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:10.036249 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:10.039015 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.039615 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.039620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.039624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.039746 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.040063 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.040068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.040072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.040331 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:10.040455 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:10.040833 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 19:39:10.041045 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 19:39:10.041822 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.029132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34743 (local address 127.2.164.65:33159)
0926 19:39:10.029721 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:10.029736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:10.029862 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:10.030196 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:10.030200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:10.030631 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:10.030896 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.030905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.031850 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.031855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.033244 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.033264 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.034235 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.034242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.034382 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.034945 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:10.034973 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:10.036005 (+ 1032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:10.039138 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.039144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.039148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.039512 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.039844 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.039849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.039853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.039971 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.040436 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:10.040441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:10.040616 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 19:39:10.040734 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 19:39:10.041032 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":308,"threads_started":1}
I20260926 19:39:10.046192 3899 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:45101
I20260926 19:39:10.046229 4033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:45101 every 8 connection(s)
I20260926 19:39:10.047289 3899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:10.049059 4034 heartbeater.cc:347] Connected to a master server at 127.2.164.126:34743
I20260926 19:39:10.049261 4034 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:10.049814 4034 heartbeater.cc:511] Master 127.2.164.126:34743 requested a full tablet report, sending...
I20260926 19:39:10.050840 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 3899
I20260926 19:39:10.051033 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:10.051625 2705 external_mini_cluster.cc:1428] Setting key 45774a5672dbda9b4841abc9f04668eb
I20260926 19:39:10.051674 3828 ts_manager.cc:194] Registered new tserver with Master: f04c989a08d2423aae06476aea8cc63a (127.2.164.65:45101)
I20260926 19:39:10.053237 3828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:38771
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:39:10.086027 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34743
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:10.382366 4038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:10.382648 4038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:10.382773 4038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:10.390580 4038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:10.390699 4038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:10.390826 4038 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:39:10.390933 4038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:10.400267 4038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:34743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:10.401448 4038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:10.402693 4038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:10.417362 4043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:10.417523 4046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:10.417627 4044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:10.418179 4038 server_base.cc:1034] running on GCE node
I20260926 19:39:10.418708 4038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:10.419517 4038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:10.420718 4038 hybrid_clock.cc:648] HybridClock initialized: now 1790451550420653 us; error 41 us; skew 500 ppm
I20260926 19:39:10.420956 4038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:10 dist-test-slave-mrw4 krb5kdc[3783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451550, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:10.423807 4038 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:39:10.425084 4038 webserver.cc:460] Webserver started at http://127.2.164.66:38973/ using document root <none> and password file <none>
I20260926 19:39:10.425721 4038 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:10.425807 4038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:10.426092 4038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:10.428721 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "acc1b16893204cc4a2bd18d64d830de0"
format_stamp: "Formatted at 2026-09-26 19:39:10 on dist-test-slave-mrw4"
server_key: "29fbd38b3fc6b215a3a726ddcafdd815"
server_key_iv: "67adb3956f8c0812394eafa34e8c9f73"
server_key_version: "encryptionkey@0"
I20260926 19:39:10.429483 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "acc1b16893204cc4a2bd18d64d830de0"
format_stamp: "Formatted at 2026-09-26 19:39:10 on dist-test-slave-mrw4"
server_key: "29fbd38b3fc6b215a3a726ddcafdd815"
server_key_iv: "67adb3956f8c0812394eafa34e8c9f73"
server_key_version: "encryptionkey@0"
I20260926 19:39:10.433960 4038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:39:10.437093 4053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:10.438527 4038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:39:10.438658 4038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "acc1b16893204cc4a2bd18d64d830de0"
format_stamp: "Formatted at 2026-09-26 19:39:10 on dist-test-slave-mrw4"
server_key: "29fbd38b3fc6b215a3a726ddcafdd815"
server_key_iv: "67adb3956f8c0812394eafa34e8c9f73"
server_key_version: "encryptionkey@0"
I20260926 19:39:10.438838 4038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:10.463106 4038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:10.466799 4038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:10.467024 4038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:10.469694 4038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:10.469785 4038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:10.469919 4038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:10.470005 4038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:39:10 dist-test-slave-mrw4 krb5kdc[3783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451550, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:10.493932 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.470125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:43093 (local address 127.2.164.126:34743)
0926 19:39:10.470341 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:10.470355 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:10.471668 (+ 1313us) server_negotiation.cc:407] Connection header received
0926 19:39:10.473688 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:10.473694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:10.473770 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:10.473857 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:10.475881 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.477020 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.479839 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.480471 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.480598 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.484896 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:10.484903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:10.484908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:10.484946 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:10.488082 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.488974 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.488977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.488980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.489121 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.489737 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.489742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.489749 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.490141 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:10.490290 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:10.493342 (+ 3052us) server_negotiation.cc:299] Negotiation successful
0926 19:39:10.493595 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 19:39:10.494376 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.470766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34743 (local address 127.2.164.66:43093)
0926 19:39:10.471520 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:10.471556 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:10.473405 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:10.474128 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:10.474140 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:10.474867 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:10.475665 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.475703 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.477248 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.477256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.479630 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.479645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.480708 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.480716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.480926 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.481606 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:10.481627 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:10.484658 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:10.488218 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.488237 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.488252 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.488790 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.489264 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.489269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.489273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.489578 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.490361 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:10.490369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:10.490846 (+ 477us) client_negotiation.cc:769] Sending connection context
0926 19:39:10.491189 (+ 343us) client_negotiation.cc:240] Negotiation successful
0926 19:39:10.491748 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":315,"threads_started":1}
I20260926 19:39:10.513089 4038 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:37551
I20260926 19:39:10.513126 4172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:37551 every 8 connection(s)
I20260926 19:39:10.514639 4038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:10.516892 4173 heartbeater.cc:347] Connected to a master server at 127.2.164.126:34743
I20260926 19:39:10.517184 4173 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:10.517817 4173 heartbeater.cc:511] Master 127.2.164.126:34743 requested a full tablet report, sending...
I20260926 19:39:10.519208 3828 ts_manager.cc:194] Registered new tserver with Master: acc1b16893204cc4a2bd18d64d830de0 (127.2.164.66:37551)
I20260926 19:39:10.519665 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 4038
I20260926 19:39:10.519845 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:10.520116 3828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:43093
I20260926 19:39:10.520339 2705 external_mini_cluster.cc:1428] Setting key 03d1f9a115ec983f898d0cf7e0d7f23f
I20260926 19:39:10.521952 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.508768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:59099 (local address 127.2.164.126:34743)
0926 19:39:10.508963 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:10.508969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:10.509063 (+ 94us) server_negotiation.cc:407] Connection header received
0926 19:39:10.509330 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:10.509337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:10.509409 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:10.509472 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:10.510739 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.511993 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.513508 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.514114 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.514246 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.516293 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:10.516300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:10.516312 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:10.516352 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:10.519059 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.519778 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.519787 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.519791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.519937 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.520322 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.520327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.520331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.520726 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:10.520879 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:10.521394 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 19:39:10.521634 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 19:39:10.522058 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.508379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34743 (local address 127.2.164.66:59099)
0926 19:39:10.508959 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:10.508979 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:10.509198 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:10.509656 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:10.509661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:10.510230 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:10.510571 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.510583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.512150 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.512154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.513378 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.513390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.514302 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.514306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.514440 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.515011 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:10.515036 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:10.516114 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:10.519176 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.519182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.519186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.519657 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.520059 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.520064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.520068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.520213 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.520928 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:10.520932 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:10.521080 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 19:39:10.521273 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 19:39:10.521621 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":582,"threads_started":1}
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:39:10.554230 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34743
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:10.845808 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:10.846063 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:10.846180 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:10.853049 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:10.853152 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:10.853284 4177 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:39:10.853389 4177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:10.862515 4177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:34743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:10.863626 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:10.864827 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:10.876878 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:10.876933 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:10.876911 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:10.877599 4177 server_base.cc:1034] running on GCE node
I20260926 19:39:10.878116 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:10.878895 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:10.880090 4177 hybrid_clock.cc:648] HybridClock initialized: now 1790451550880032 us; error 41 us; skew 500 ppm
I20260926 19:39:10.880344 4177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:10 dist-test-slave-mrw4 krb5kdc[3783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451550, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:10.883134 4177 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:39:10.884394 4177 webserver.cc:460] Webserver started at http://127.2.164.67:34831/ using document root <none> and password file <none>
I20260926 19:39:10.885008 4177 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:10.885097 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:10.885409 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:10.887995 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2"
format_stamp: "Formatted at 2026-09-26 19:39:10 on dist-test-slave-mrw4"
server_key: "28a2931d80fca5e35e5268265eeceff9"
server_key_iv: "21dbe6b6aa82eb28d4fc14ec15e6dcba"
server_key_version: "encryptionkey@0"
I20260926 19:39:10.888708 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2"
format_stamp: "Formatted at 2026-09-26 19:39:10 on dist-test-slave-mrw4"
server_key: "28a2931d80fca5e35e5268265eeceff9"
server_key_iv: "21dbe6b6aa82eb28d4fc14ec15e6dcba"
server_key_version: "encryptionkey@0"
I20260926 19:39:10.893240 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 19:39:10.896287 4192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:10.897516 4177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:39:10.897660 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2"
format_stamp: "Formatted at 2026-09-26 19:39:10 on dist-test-slave-mrw4"
server_key: "28a2931d80fca5e35e5268265eeceff9"
server_key_iv: "21dbe6b6aa82eb28d4fc14ec15e6dcba"
server_key_version: "encryptionkey@0"
I20260926 19:39:10.897835 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:10.927537 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:10.931267 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:10.931504 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:10.934028 4177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:10.934104 4177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:10.934237 4177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:10.934335 4177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 26 19:39:10 dist-test-slave-mrw4 krb5kdc[3783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451550, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:10.956248 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.934522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:40399 (local address 127.2.164.126:34743)
0926 19:39:10.934700 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:10.934706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:10.936052 (+ 1346us) server_negotiation.cc:407] Connection header received
0926 19:39:10.938022 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:10.938029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:10.938120 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:10.938228 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:10.940508 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.941522 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.944282 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.944839 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.944984 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.949461 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:10.949467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:10.949472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:10.949506 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:10.952804 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.953861 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.953864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.953867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.954035 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.954493 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.954498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.954502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.954809 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:10.954965 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:10.955808 (+ 843us) server_negotiation.cc:299] Negotiation successful
0926 19:39:10.955978 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:39:10.957839 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.935046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34743 (local address 127.2.164.67:40399)
0926 19:39:10.935883 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:10.935929 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:10.937755 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:10.938467 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:10.938482 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:10.939467 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:10.940327 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.940357 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.941689 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.941695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.944120 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.944137 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.945083 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.945090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.945410 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.946128 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:10.946151 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:10.949244 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:10.953030 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.953039 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.953062 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.953707 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.954174 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.954179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.954183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.954378 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.954964 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:10.954981 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:10.955426 (+ 445us) client_negotiation.cc:769] Sending connection context
0926 19:39:10.955722 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 19:39:10.956289 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":326,"threads_started":1}
I20260926 19:39:10.976362 4276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.963007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34743 (local address 127.2.164.67:51701)
0926 19:39:10.963527 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:10.963546 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:10.963664 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:10.963935 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:10.963939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:10.964328 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:10.964584 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.964592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.965721 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.965728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.967687 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:10.967706 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.968515 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.968522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:10.968691 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.969350 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:10.969373 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:10.970425 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:10.973773 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.973778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.973782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.974272 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.974686 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:10.974692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:10.974697 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:10.974863 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:10.975527 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:10.975533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:10.975700 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 19:39:10.975824 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 19:39:10.976042 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":277,"threads_started":1}
I20260926 19:39:10.976542 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:10.962990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:51701 (local address 127.2.164.126:34743)
0926 19:39:10.963129 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:10.963133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:10.963624 (+ 491us) server_negotiation.cc:407] Connection header received
0926 19:39:10.963760 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:10.963764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:10.963821 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:10.963895 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:10.964707 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.965566 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.967833 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:10.968338 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:10.968439 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:10.970611 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:10.970615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:10.970618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:10.970647 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:10.973643 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.974389 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.974394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.974398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.974548 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:10.975013 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:10.975018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:10.975021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:10.975391 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:10.975547 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:10.975941 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 19:39:10.976172 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:39:10.979797 4177 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:45717
I20260926 19:39:10.979823 4311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:45717 every 8 connection(s)
I20260926 19:39:10.981029 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:10.982802 4312 heartbeater.cc:347] Connected to a master server at 127.2.164.126:34743
I20260926 19:39:10.983012 4312 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:10.983636 4312 heartbeater.cc:511] Master 127.2.164.126:34743 requested a full tablet report, sending...
I20260926 19:39:10.984802 3828 ts_manager.cc:194] Registered new tserver with Master: 9ebb92e730eb42b8b96d4f1f7afd99f2 (127.2.164.67:45717)
I20260926 19:39:10.985639 3828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:40399
I20260926 19:39:10.986961 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 4177
I20260926 19:39:10.987108 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:10.987495 2705 external_mini_cluster.cc:1428] Setting key 0288b937aad68fc97478420c74c6c5d3
I20260926 19:39:10.990195 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:39:11 dist-test-slave-mrw4 krb5kdc[3783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 19:39:11 dist-test-slave-mrw4 krb5kdc[3783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:11.031291 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.018050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.2.164.126:34743)
0926 19:39:11.018268 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.018275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.018505 (+ 230us) server_negotiation.cc:407] Connection header received
0926 19:39:11.018688 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.018692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.018777 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.018878 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:11.020138 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.020914 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.022146 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.022769 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.022874 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.026585 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:11.026590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:11.026593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:11.026626 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:11.029015 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.029664 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.029668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.029671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.029808 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.030161 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.030164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.030167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.030418 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:11.030587 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.030901 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.031059 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:39:11.035116 3828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:39:11.041921 3828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:11.055725 4034 heartbeater.cc:503] Master 127.2.164.126:34743 was elected leader, sending a full tablet report...
I20260926 19:39:11.063625 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.053754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.2.164.65:45101)
0926 19:39:11.054272 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.054281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.054823 (+ 542us) server_negotiation.cc:407] Connection header received
0926 19:39:11.055217 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.055226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.055483 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.055677 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:11.056665 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.058344 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.061203 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.062502 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.062626 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.062863 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.063173 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.063342 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":245,"threads_started":1}
I20260926 19:39:11.063733 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.053506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:45101 (local address 127.0.0.1:44964)
0926 19:39:11.054689 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:11.054729 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:11.054980 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:11.055993 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:11.056001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:11.056047 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:11.056482 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.056495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.058558 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.058563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.061002 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.061015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.062682 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.062688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.062895 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.062973 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 19:39:11.063120 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 19:39:11.063307 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":4,"spinlock_wait_cycles":14976,"thread_start_us":795,"threads_started":1}
I20260926 19:39:11.066771 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.054710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.2.164.66:37551)
0926 19:39:11.059210 (+ 4500us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.059220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.059252 (+ 32us) server_negotiation.cc:407] Connection header received
0926 19:39:11.059349 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.059357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.059640 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.059825 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:11.060471 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.061847 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.064556 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.065822 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.065945 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.066131 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.066361 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.066523 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":532,"threads_started":1}
I20260926 19:39:11.066799 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.054219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:37551 (local address 127.0.0.1:35758)
0926 19:39:11.054786 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:11.054804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:11.054980 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:11.059808 (+ 4828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:11.059812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:11.059837 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:11.060284 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.060296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.061972 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.061977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.064403 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.064416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.066018 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.066024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.066182 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.066208 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 19:39:11.066293 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 19:39:11.066445 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":23,"spinlock_wait_cycles":3968,"thread_start_us":623,"threads_started":1}
I20260926 19:39:11.067674 3964 tablet_service.cc:1467] Processing CreateTablet for tablet 7fab135443874b53ad3811fc68284ec1 (DEFAULT_TABLE table=test-table [id=b9349e93cde444a1a1cf12dd3388cf8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:11.068526 4329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.055107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:45717 (local address 127.0.0.1:51838)
0926 19:39:11.061048 (+ 5941us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:11.061066 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:11.061260 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:11.062055 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:11.062058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:11.062080 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:11.062402 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.062410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.064218 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.064225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.066083 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.066097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.067785 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.067791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.067947 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.067978 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:39:11.068070 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 19:39:11.068186 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5817,"thread_start_us":5715,"threads_started":1}
I20260926 19:39:11.068573 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.055612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.2.164.67:45717)
0926 19:39:11.060479 (+ 4867us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.060486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.061148 (+ 662us) server_negotiation.cc:407] Connection header received
0926 19:39:11.061434 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.061442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.061654 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.061783 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:11.062600 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.064104 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.066273 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.067597 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.067715 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.067932 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.068135 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.068279 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4696,"thread_start_us":4898,"threads_started":1}
I20260926 19:39:11.069304 3964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fab135443874b53ad3811fc68284ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:11.070329 4103 tablet_service.cc:1467] Processing CreateTablet for tablet 7fab135443874b53ad3811fc68284ec1 (DEFAULT_TABLE table=test-table [id=b9349e93cde444a1a1cf12dd3388cf8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:11.071633 4103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fab135443874b53ad3811fc68284ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:11.072110 4242 tablet_service.cc:1467] Processing CreateTablet for tablet 7fab135443874b53ad3811fc68284ec1 (DEFAULT_TABLE table=test-table [id=b9349e93cde444a1a1cf12dd3388cf8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:11.073402 4242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fab135443874b53ad3811fc68284ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:11.080776 4331 tablet_bootstrap.cc:492] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a: Bootstrap starting.
I20260926 19:39:11.084012 4331 tablet_bootstrap.cc:654] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:11.084934 4332 tablet_bootstrap.cc:492] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0: Bootstrap starting.
I20260926 19:39:11.085186 4331 log.cc:824] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:11.088811 4333 tablet_bootstrap.cc:492] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Bootstrap starting.
I20260926 19:39:11.088972 4332 tablet_bootstrap.cc:654] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:11.090297 4332 log.cc:824] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:11.091588 4333 tablet_bootstrap.cc:654] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:11.092604 4333 log.cc:824] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:11.093029 4332 tablet_bootstrap.cc:492] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0: No bootstrap required, opened a new log
I20260926 19:39:11.093319 4332 ts_tablet_manager.cc:1397] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 19:39:11.094139 4331 tablet_bootstrap.cc:492] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a: No bootstrap required, opened a new log
I20260926 19:39:11.094383 4331 ts_tablet_manager.cc:1397] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20260926 19:39:11.094998 4333 tablet_bootstrap.cc:492] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2: No bootstrap required, opened a new log
I20260926 19:39:11.095222 4333 ts_tablet_manager.cc:1397] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 19:39:11.102058 4333 raft_consensus.cc:348] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.102382 4333 raft_consensus.cc:374] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:11.102478 4333 raft_consensus.cc:729] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb92e730eb42b8b96d4f1f7afd99f2, State: Initialized, Role: FOLLOWER
I20260926 19:39:11.102980 4333 consensus_queue.cc:260] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.104074 4331 raft_consensus.cc:348] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.103808 4332 raft_consensus.cc:348] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.104559 4331 raft_consensus.cc:374] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:11.104574 4332 raft_consensus.cc:374] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:11.104660 4331 raft_consensus.cc:729] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04c989a08d2423aae06476aea8cc63a, State: Initialized, Role: FOLLOWER
I20260926 19:39:11.104660 4332 raft_consensus.cc:729] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc1b16893204cc4a2bd18d64d830de0, State: Initialized, Role: FOLLOWER
I20260926 19:39:11.105211 4331 consensus_queue.cc:260] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.105216 4332 consensus_queue.cc:260] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.110112 4332 ts_tablet_manager.cc:1428] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0: Time spent starting tablet: real 0.017s user 0.007s sys 0.007s
I20260926 19:39:11.110520 4173 heartbeater.cc:503] Master 127.2.164.126:34743 was elected leader, sending a full tablet report...
I20260926 19:39:11.112488 4312 heartbeater.cc:503] Master 127.2.164.126:34743 was elected leader, sending a full tablet report...
I20260926 19:39:11.112435 4338 raft_consensus.cc:482] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:11.112699 4338 raft_consensus.cc:504] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.114665 4331 ts_tablet_manager.cc:1428] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a: Time spent starting tablet: real 0.020s user 0.010s sys 0.011s
I20260926 19:39:11.114457 4333 ts_tablet_manager.cc:1428] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
I20260926 19:39:11.116809 4338 leader_election.cc:288] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f04c989a08d2423aae06476aea8cc63a (127.2.164.65:45101), acc1b16893204cc4a2bd18d64d830de0 (127.2.164.66:37551)
I20260926 19:39:11.123172 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.117052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:37551 (local address 127.2.164.67:34979)
0926 19:39:11.117547 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:11.117564 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:11.117677 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:11.118174 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:11.118179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:11.118204 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:11.118378 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.118386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.119651 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.119655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.121386 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.121397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.122529 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.122535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.122712 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.122743 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:39:11.122827 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 19:39:11.122955 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398}
I20260926 19:39:11.123242 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.117352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:34979 (local address 127.2.164.66:37551)
0926 19:39:11.117511 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.117516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.117802 (+ 286us) server_negotiation.cc:407] Connection header received
0926 19:39:11.117862 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.117867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.117937 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.118015 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:11.118506 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.119513 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.121522 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.122394 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.122510 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.122559 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.122865 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.123007 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:39:11.123587 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.117242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:35117 (local address 127.2.164.65:45101)
0926 19:39:11.117408 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.117415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.118015 (+ 600us) server_negotiation.cc:407] Connection header received
0926 19:39:11.118270 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.118277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.118340 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.118399 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:11.119172 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.120040 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.121906 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.122743 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.122888 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.122959 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.123272 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.123389 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 19:39:11.123700 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.117238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:45101 (local address 127.2.164.67:35117)
0926 19:39:11.117922 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:11.117948 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:11.118108 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:11.118573 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:11.118577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:11.118598 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:11.118988 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.119001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.120164 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.120169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.121774 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.121786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.122941 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.122947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.123108 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.123141 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:39:11.123234 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 19:39:11.123352 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":748,"threads_started":1}
I20260926 19:39:11.124327 4123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fab135443874b53ad3811fc68284ec1" candidate_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc1b16893204cc4a2bd18d64d830de0" is_pre_election: true
I20260926 19:39:11.124594 3984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fab135443874b53ad3811fc68284ec1" candidate_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04c989a08d2423aae06476aea8cc63a" is_pre_election: true
I20260926 19:39:11.124718 4123 raft_consensus.cc:2454] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb92e730eb42b8b96d4f1f7afd99f2 in term 0.
I20260926 19:39:11.124967 3984 raft_consensus.cc:2454] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb92e730eb42b8b96d4f1f7afd99f2 in term 0.
I20260926 19:39:11.125279 4193 leader_election.cc:302] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb92e730eb42b8b96d4f1f7afd99f2, acc1b16893204cc4a2bd18d64d830de0; no voters:
I20260926 19:39:11.125620 4338 raft_consensus.cc:2781] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:11.125730 4338 raft_consensus.cc:482] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:11.125819 4338 raft_consensus.cc:3037] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:11.127907 4338 raft_consensus.cc:504] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.128508 4338 leader_election.cc:288] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [CANDIDATE]: Term 1 election: Requested vote from peers f04c989a08d2423aae06476aea8cc63a (127.2.164.65:45101), acc1b16893204cc4a2bd18d64d830de0 (127.2.164.66:37551)
I20260926 19:39:11.128795 3984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fab135443874b53ad3811fc68284ec1" candidate_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04c989a08d2423aae06476aea8cc63a"
I20260926 19:39:11.128991 3984 raft_consensus.cc:3037] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:11.128969 4123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fab135443874b53ad3811fc68284ec1" candidate_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc1b16893204cc4a2bd18d64d830de0"
I20260926 19:39:11.129133 4123 raft_consensus.cc:3037] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:11.131933 3984 raft_consensus.cc:2454] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb92e730eb42b8b96d4f1f7afd99f2 in term 1.
I20260926 19:39:11.131945 4123 raft_consensus.cc:2454] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb92e730eb42b8b96d4f1f7afd99f2 in term 1.
I20260926 19:39:11.132310 4194 leader_election.cc:302] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9ebb92e730eb42b8b96d4f1f7afd99f2, acc1b16893204cc4a2bd18d64d830de0, f04c989a08d2423aae06476aea8cc63a; no voters:
I20260926 19:39:11.132619 4338 raft_consensus.cc:2781] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:11.132777 4338 raft_consensus.cc:686] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebb92e730eb42b8b96d4f1f7afd99f2, State: Running, Role: LEADER
I20260926 19:39:11.133189 4338 consensus_queue.cc:237] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.136758 3827 catalog_manager.cc:5495] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ebb92e730eb42b8b96d4f1f7afd99f2 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } health_report { overall_health: UNKNOWN } } }
I20260926 19:39:11.159910 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.154960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.2.164.67:45717)
0926 19:39:11.155145 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.155149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.155167 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:39:11.155431 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.155435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.155506 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.155614 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:11.155899 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.156565 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.157906 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.158316 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.158410 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.158818 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:11.159333 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:11.159439 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.159585 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.159648 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:39:11.166985 4123 raft_consensus.cc:1261] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0 [term 1 FOLLOWER]: Refusing update from remote peer 9ebb92e730eb42b8b96d4f1f7afd99f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:11.167011 3984 raft_consensus.cc:1261] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a [term 1 FOLLOWER]: Refusing update from remote peer 9ebb92e730eb42b8b96d4f1f7afd99f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:11.167832 4337 consensus_queue.cc:1035] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:11.168128 4338 consensus_queue.cc:1035] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:11.178574 4346 mvcc.cc:205] Tried to move back new op lower bound from 7333689553571508224 to 7333689553444659200. Current Snapshot: MvccSnapshot[applied={T|T < 7333689553571508224}]
I20260926 19:39:11.197053 3827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:11.197338 3827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:11.200708 3827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527: Sending DeleteTablet for 3 replicas of tablet 7fab135443874b53ad3811fc68284ec1
I20260926 19:39:11.201797 3964 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fab135443874b53ad3811fc68284ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:11 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:44964
I20260926 19:39:11.201960 4103 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fab135443874b53ad3811fc68284ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:11 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:35758
I20260926 19:39:11.202162 4242 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fab135443874b53ad3811fc68284ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:11 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:51838
I20260926 19:39:11.204597 3827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:39:11.205430 3827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:11.214056 4102 tablet_service.cc:1467] Processing CreateTablet for tablet aa6df1b2b65d48cb9ef91247aad4c4c8 (DEFAULT_TABLE table=test-table [id=3190d144bd86437887e08c3ea722d061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:11.214162 4241 tablet_service.cc:1467] Processing CreateTablet for tablet aa6df1b2b65d48cb9ef91247aad4c4c8 (DEFAULT_TABLE table=test-table [id=3190d144bd86437887e08c3ea722d061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:11.214205 3963 tablet_service.cc:1467] Processing CreateTablet for tablet aa6df1b2b65d48cb9ef91247aad4c4c8 (DEFAULT_TABLE table=test-table [id=3190d144bd86437887e08c3ea722d061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:11.214679 4102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6df1b2b65d48cb9ef91247aad4c4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:11.214681 4241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6df1b2b65d48cb9ef91247aad4c4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:11.214699 3963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa6df1b2b65d48cb9ef91247aad4c4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:11.217738 4355 tablet_replica.cc:330] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a: stopping tablet replica
I20260926 19:39:11.218331 4355 raft_consensus.cc:2229] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:11.218925 4355 raft_consensus.cc:2258] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:11.220765 4333 tablet_bootstrap.cc:492] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Bootstrap starting.
I20260926 19:39:11.221060 4355 ts_tablet_manager.cc:1905] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:11.221362 4331 tablet_bootstrap.cc:492] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a: Bootstrap starting.
I20260926 19:39:11.221388 4332 tablet_bootstrap.cc:492] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0: Bootstrap starting.
I20260926 19:39:11.222947 4331 tablet_bootstrap.cc:654] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:11.222966 4332 tablet_bootstrap.cc:654] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:11.225277 4333 tablet_bootstrap.cc:654] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:11.229557 4357 tablet_replica.cc:330] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2: stopping tablet replica
I20260926 19:39:11.230031 4357 raft_consensus.cc:2229] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:39:11.230715 4357 raft_consensus.cc:2258] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:11.233134 4357 ts_tablet_manager.cc:1905] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:11.237358 4333 tablet_bootstrap.cc:492] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2: No bootstrap required, opened a new log
I20260926 19:39:11.237500 4333 ts_tablet_manager.cc:1397] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20260926 19:39:11.238286 4333 raft_consensus.cc:348] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } }
I20260926 19:39:11.238440 4331 tablet_bootstrap.cc:492] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a: No bootstrap required, opened a new log
I20260926 19:39:11.238572 4331 ts_tablet_manager.cc:1397] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20260926 19:39:11.239307 4331 raft_consensus.cc:348] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } }
I20260926 19:39:11.239513 4331 raft_consensus.cc:374] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:11.239589 4331 raft_consensus.cc:729] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04c989a08d2423aae06476aea8cc63a, State: Initialized, Role: FOLLOWER
I20260926 19:39:11.239591 4333 raft_consensus.cc:374] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:11.239699 4333 raft_consensus.cc:729] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb92e730eb42b8b96d4f1f7afd99f2, State: Initialized, Role: FOLLOWER
I20260926 19:39:11.239959 4331 consensus_queue.cc:260] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } }
I20260926 19:39:11.240049 4333 consensus_queue.cc:260] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } }
I20260926 19:39:11.240463 4332 tablet_bootstrap.cc:492] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0: No bootstrap required, opened a new log
I20260926 19:39:11.240602 4332 ts_tablet_manager.cc:1397] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.001s
I20260926 19:39:11.241460 4332 raft_consensus.cc:348] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } }
I20260926 19:39:11.241671 4332 raft_consensus.cc:374] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:11.241789 4332 raft_consensus.cc:729] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc1b16893204cc4a2bd18d64d830de0, State: Initialized, Role: FOLLOWER
I20260926 19:39:11.242093 4331 ts_tablet_manager.cc:1428] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 19:39:11.242019 4332 consensus_queue.cc:260] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } }
I20260926 19:39:11.242558 4332 ts_tablet_manager.cc:1428] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 19:39:11.243108 4355 ts_tablet_manager.cc:1918] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:11.243244 4355 log.cc:1196] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/7fab135443874b53ad3811fc68284ec1
I20260926 19:39:11.243726 4355 ts_tablet_manager.cc:1939] T 7fab135443874b53ad3811fc68284ec1 P f04c989a08d2423aae06476aea8cc63a: Deleting consensus metadata
I20260926 19:39:11.245407 3813 catalog_manager.cc:4841] TS f04c989a08d2423aae06476aea8cc63a (127.2.164.65:45101): tablet 7fab135443874b53ad3811fc68284ec1 (table test-table [id=b9349e93cde444a1a1cf12dd3388cf8b]) successfully deleted
I20260926 19:39:11.247956 4333 ts_tablet_manager.cc:1428] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260926 19:39:11.250440 4356 tablet_replica.cc:330] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0: stopping tablet replica
I20260926 19:39:11.250950 4356 raft_consensus.cc:2229] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:11.251749 4356 raft_consensus.cc:2258] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:11.254614 4357 ts_tablet_manager.cc:1918] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:11.254673 4356 ts_tablet_manager.cc:1905] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:11.254850 4357 log.cc:1196] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/7fab135443874b53ad3811fc68284ec1
I20260926 19:39:11.255417 4357 ts_tablet_manager.cc:1939] T 7fab135443874b53ad3811fc68284ec1 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Deleting consensus metadata
I20260926 19:39:11.257501 3813 catalog_manager.cc:4841] TS 9ebb92e730eb42b8b96d4f1f7afd99f2 (127.2.164.67:45717): tablet 7fab135443874b53ad3811fc68284ec1 (table test-table [id=b9349e93cde444a1a1cf12dd3388cf8b]) successfully deleted
I20260926 19:39:11.260241 4356 ts_tablet_manager.cc:1918] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:11.260365 4356 log.cc:1196] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/7fab135443874b53ad3811fc68284ec1
I20260926 19:39:11.260833 4356 ts_tablet_manager.cc:1939] T 7fab135443874b53ad3811fc68284ec1 P acc1b16893204cc4a2bd18d64d830de0: Deleting consensus metadata
I20260926 19:39:11.262283 3812 catalog_manager.cc:4841] TS acc1b16893204cc4a2bd18d64d830de0 (127.2.164.66:37551): tablet 7fab135443874b53ad3811fc68284ec1 (table test-table [id=b9349e93cde444a1a1cf12dd3388cf8b]) successfully deleted
W20260926 19:39:11.266703 4174 tablet.cc:2369] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:11.291019 4340 raft_consensus.cc:482] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:11.291245 4340 raft_consensus.cc:504] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } }
I20260926 19:39:11.293128 4340 leader_election.cc:288] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc1b16893204cc4a2bd18d64d830de0 (127.2.164.66:37551), 9ebb92e730eb42b8b96d4f1f7afd99f2 (127.2.164.67:45717)
W20260926 19:39:11.299351 4035 tablet.cc:2369] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:11.300935 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.293603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:32863 (local address 127.2.164.67:45717)
0926 19:39:11.293762 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.293767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.294302 (+ 535us) server_negotiation.cc:407] Connection header received
0926 19:39:11.294576 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.294582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.294656 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.294736 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:11.295693 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.296688 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.298560 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.299623 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.299771 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.299863 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.300503 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.300646 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:39:11.301110 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.293397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:59665 (local address 127.2.164.66:37551)
0926 19:39:11.293571 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.293578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.294628 (+ 1050us) server_negotiation.cc:407] Connection header received
0926 19:39:11.294769 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.294774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.294846 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.294942 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:11.295400 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.296417 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.298987 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.300082 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.300221 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.300288 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.300720 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.300840 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:39:11.301614 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.293524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:45717 (local address 127.2.164.65:32863)
0926 19:39:11.294206 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:11.294228 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:11.294417 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:11.295112 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:11.295117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:11.295142 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:11.295519 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.295530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.296803 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.296808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.298414 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.298425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.300157 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.300164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.300322 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.300354 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 19:39:11.301091 (+ 737us) client_negotiation.cc:240] Negotiation successful
0926 19:39:11.301242 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":23,"thread_start_us":394,"threads_started":1}
I20260926 19:39:11.301878 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.293740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:37551 (local address 127.2.164.65:59665)
0926 19:39:11.294515 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:11.294531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:11.294658 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:11.294959 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:11.294962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:11.294981 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:11.295257 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.295266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.296548 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.296554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.298836 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.298851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.300419 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.300425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.300566 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.300597 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:39:11.301360 (+ 763us) client_negotiation.cc:240] Negotiation successful
0926 19:39:11.301476 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":4,"thread_start_us":504,"threads_started":1}
I20260926 19:39:11.302415 4123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6df1b2b65d48cb9ef91247aad4c4c8" candidate_uuid: "f04c989a08d2423aae06476aea8cc63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc1b16893204cc4a2bd18d64d830de0" is_pre_election: true
I20260926 19:39:11.302459 4262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6df1b2b65d48cb9ef91247aad4c4c8" candidate_uuid: "f04c989a08d2423aae06476aea8cc63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" is_pre_election: true
I20260926 19:39:11.302656 4123 raft_consensus.cc:2454] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f04c989a08d2423aae06476aea8cc63a in term 0.
I20260926 19:39:11.302863 4262 raft_consensus.cc:2454] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f04c989a08d2423aae06476aea8cc63a in term 0.
I20260926 19:39:11.303148 3915 leader_election.cc:302] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc1b16893204cc4a2bd18d64d830de0, f04c989a08d2423aae06476aea8cc63a; no voters:
I20260926 19:39:11.303506 4340 raft_consensus.cc:2781] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:11.303611 4340 raft_consensus.cc:482] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:11.303699 4340 raft_consensus.cc:3037] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:11.306712 4340 raft_consensus.cc:504] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } }
I20260926 19:39:11.307355 4340 leader_election.cc:288] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [CANDIDATE]: Term 1 election: Requested vote from peers acc1b16893204cc4a2bd18d64d830de0 (127.2.164.66:37551), 9ebb92e730eb42b8b96d4f1f7afd99f2 (127.2.164.67:45717)
I20260926 19:39:11.307729 4262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6df1b2b65d48cb9ef91247aad4c4c8" candidate_uuid: "f04c989a08d2423aae06476aea8cc63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2"
I20260926 19:39:11.307729 4123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6df1b2b65d48cb9ef91247aad4c4c8" candidate_uuid: "f04c989a08d2423aae06476aea8cc63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc1b16893204cc4a2bd18d64d830de0"
I20260926 19:39:11.307886 4262 raft_consensus.cc:3037] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:11.307904 4123 raft_consensus.cc:3037] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:11.310045 4262 raft_consensus.cc:2454] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f04c989a08d2423aae06476aea8cc63a in term 1.
I20260926 19:39:11.310393 3916 leader_election.cc:302] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb92e730eb42b8b96d4f1f7afd99f2, f04c989a08d2423aae06476aea8cc63a; no voters:
I20260926 19:39:11.310647 4340 raft_consensus.cc:2781] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:11.310663 4123 raft_consensus.cc:2454] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f04c989a08d2423aae06476aea8cc63a in term 1.
I20260926 19:39:11.311410 4340 raft_consensus.cc:686] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 1 LEADER]: Becoming Leader. State: Replica: f04c989a08d2423aae06476aea8cc63a, State: Running, Role: LEADER
I20260926 19:39:11.311793 4340 consensus_queue.cc:237] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } }
I20260926 19:39:11.315148 3827 catalog_manager.cc:5495] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a reported cstate change: term changed from 0 to 1, leader changed from <none> to f04c989a08d2423aae06476aea8cc63a (127.2.164.65). New cstate: current_term: 1 leader_uuid: "f04c989a08d2423aae06476aea8cc63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } health_report { overall_health: UNKNOWN } } }
Sep 26 19:39:11 dist-test-slave-mrw4 krb5kdc[3783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451549, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:11.384239 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.367359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.2.164.126:34743)
0926 19:39:11.367531 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.367537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.368098 (+ 561us) server_negotiation.cc:407] Connection header received
0926 19:39:11.368237 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.368243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.368319 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.368432 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:11.369583 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.370726 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.371924 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.372579 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.372702 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.379159 (+ 6457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:11.379164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:11.379168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:11.379194 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:11.381632 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.382435 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.382438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.382441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.382582 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.382956 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.382959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.382961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.383194 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:11.383296 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.383752 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.383905 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:39:11.384529 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.367407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34743 (local address 127.0.0.1:44032)
0926 19:39:11.367989 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:11.368009 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:11.368133 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:11.368475 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:11.368480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:11.369139 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:11.369426 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.369444 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.370840 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.370844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.371782 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.371793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.372739 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.372743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.372859 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.376669 (+ 3810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:11.376686 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:11.378988 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:11.381745 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:11.381757 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:11.381766 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:11.382336 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:11.382677 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:11.382682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:11.382686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:11.382847 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:11.383305 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:11.383314 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:11.383501 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 19:39:11.383686 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 19:39:11.384062 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":105,"thread_start_us":326,"threads_started":1}
I20260926 19:39:11.389351 3827 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:44032:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 19:39:11.390568 3827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:11.396997 3963 tablet_service.cc:1467] Processing CreateTablet for tablet b27612364cb74ac5b9b8386a9dd3d5f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1073835d40d844dc8b7fd6baf38a2597]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:39:11.397341 4103 tablet_service.cc:1467] Processing CreateTablet for tablet b27612364cb74ac5b9b8386a9dd3d5f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1073835d40d844dc8b7fd6baf38a2597]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:39:11.397542 3963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27612364cb74ac5b9b8386a9dd3d5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:11.397811 4103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27612364cb74ac5b9b8386a9dd3d5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:11.401783 4242 tablet_service.cc:1467] Processing CreateTablet for tablet b27612364cb74ac5b9b8386a9dd3d5f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1073835d40d844dc8b7fd6baf38a2597]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:39:11.402275 4242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27612364cb74ac5b9b8386a9dd3d5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:11.403321 4332 tablet_bootstrap.cc:492] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0: Bootstrap starting.
I20260926 19:39:11.404843 4332 tablet_bootstrap.cc:654] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:11.407626 4333 tablet_bootstrap.cc:492] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Bootstrap starting.
I20260926 19:39:11.408595 4331 tablet_bootstrap.cc:492] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: Bootstrap starting.
I20260926 19:39:11.409236 4333 tablet_bootstrap.cc:654] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:11.410145 4331 tablet_bootstrap.cc:654] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:11.410164 4332 tablet_bootstrap.cc:492] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0: No bootstrap required, opened a new log
I20260926 19:39:11.411260 4332 ts_tablet_manager.cc:1397] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260926 19:39:11.412024 4332 raft_consensus.cc:348] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.412288 4332 raft_consensus.cc:374] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:11.412364 4332 raft_consensus.cc:729] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc1b16893204cc4a2bd18d64d830de0, State: Initialized, Role: FOLLOWER
I20260926 19:39:11.412675 4332 consensus_queue.cc:260] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.413249 4332 ts_tablet_manager.cc:1428] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:11.413107 4339 tablet_replica.cc:439] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } }
I20260926 19:39:11.413970 4339 tablet_replica.cc:442] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:11.415421 4331 tablet_bootstrap.cc:492] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: No bootstrap required, opened a new log
I20260926 19:39:11.415611 4333 tablet_bootstrap.cc:492] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2: No bootstrap required, opened a new log
I20260926 19:39:11.415692 4331 ts_tablet_manager.cc:1397] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260926 19:39:11.415817 4333 ts_tablet_manager.cc:1397] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 19:39:11.416510 4333 raft_consensus.cc:348] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.416500 4331 raft_consensus.cc:348] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.416738 4333 raft_consensus.cc:374] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:11.416822 4333 raft_consensus.cc:729] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb92e730eb42b8b96d4f1f7afd99f2, State: Initialized, Role: FOLLOWER
I20260926 19:39:11.416982 4331 raft_consensus.cc:374] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:11.417104 4331 raft_consensus.cc:729] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04c989a08d2423aae06476aea8cc63a, State: Initialized, Role: FOLLOWER
I20260926 19:39:11.417062 4333 consensus_queue.cc:260] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.417423 4331 consensus_queue.cc:260] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.417472 4337 tablet_replica.cc:439] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } }
I20260926 19:39:11.417776 4337 tablet_replica.cc:442] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:11.417943 4340 tablet_replica.cc:439] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } }
I20260926 19:39:11.418188 4340 tablet_replica.cc:442] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:11.418224 4333 ts_tablet_manager.cc:1428] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:11.418705 4331 ts_tablet_manager.cc:1428] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260926 19:39:11.465641 4340 raft_consensus.cc:482] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:11.465821 4340 raft_consensus.cc:504] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.466429 4340 leader_election.cc:288] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ebb92e730eb42b8b96d4f1f7afd99f2 (127.2.164.67:45717), acc1b16893204cc4a2bd18d64d830de0 (127.2.164.66:37551)
I20260926 19:39:11.466926 4262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27612364cb74ac5b9b8386a9dd3d5f3" candidate_uuid: "f04c989a08d2423aae06476aea8cc63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" is_pre_election: true
I20260926 19:39:11.467006 4123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27612364cb74ac5b9b8386a9dd3d5f3" candidate_uuid: "f04c989a08d2423aae06476aea8cc63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc1b16893204cc4a2bd18d64d830de0" is_pre_election: true
I20260926 19:39:11.467200 4262 raft_consensus.cc:2454] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f04c989a08d2423aae06476aea8cc63a in term 0.
I20260926 19:39:11.467228 4123 raft_consensus.cc:2454] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f04c989a08d2423aae06476aea8cc63a in term 0.
I20260926 19:39:11.467634 3915 leader_election.cc:302] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc1b16893204cc4a2bd18d64d830de0, f04c989a08d2423aae06476aea8cc63a; no voters:
I20260926 19:39:11.467969 4340 raft_consensus.cc:2781] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:11.468084 4340 raft_consensus.cc:482] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:11.468194 4340 raft_consensus.cc:3037] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:11.471191 4340 raft_consensus.cc:504] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.471800 4340 leader_election.cc:288] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [CANDIDATE]: Term 1 election: Requested vote from peers 9ebb92e730eb42b8b96d4f1f7afd99f2 (127.2.164.67:45717), acc1b16893204cc4a2bd18d64d830de0 (127.2.164.66:37551)
I20260926 19:39:11.472184 4262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27612364cb74ac5b9b8386a9dd3d5f3" candidate_uuid: "f04c989a08d2423aae06476aea8cc63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2"
I20260926 19:39:11.472188 4123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27612364cb74ac5b9b8386a9dd3d5f3" candidate_uuid: "f04c989a08d2423aae06476aea8cc63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc1b16893204cc4a2bd18d64d830de0"
I20260926 19:39:11.472388 4123 raft_consensus.cc:3037] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:11.472404 4262 raft_consensus.cc:3037] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:11.474319 4123 raft_consensus.cc:2454] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f04c989a08d2423aae06476aea8cc63a in term 1.
I20260926 19:39:11.474676 3915 leader_election.cc:302] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc1b16893204cc4a2bd18d64d830de0, f04c989a08d2423aae06476aea8cc63a; no voters:
I20260926 19:39:11.474912 4340 raft_consensus.cc:2781] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:11.475072 4340 raft_consensus.cc:686] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [term 1 LEADER]: Becoming Leader. State: Replica: f04c989a08d2423aae06476aea8cc63a, State: Running, Role: LEADER
I20260926 19:39:11.475327 4262 raft_consensus.cc:2454] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f04c989a08d2423aae06476aea8cc63a in term 1.
I20260926 19:39:11.475338 4340 consensus_queue.cc:237] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } }
I20260926 19:39:11.476967 4363 tablet_replica.cc:439] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: TxnStatusTablet state changed. Reason: New leader f04c989a08d2423aae06476aea8cc63a. Latest consensus state: current_term: 1 leader_uuid: "f04c989a08d2423aae06476aea8cc63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } }
I20260926 19:39:11.477367 4363 tablet_replica.cc:442] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:39:11.478008 4375 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 19:39:11.478195 4375 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 19:39:11.479115 3827 catalog_manager.cc:5495] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a reported cstate change: term changed from 0 to 1, leader changed from <none> to f04c989a08d2423aae06476aea8cc63a (127.2.164.65). New cstate: current_term: 1 leader_uuid: "f04c989a08d2423aae06476aea8cc63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } health_report { overall_health: UNKNOWN } } }
W20260926 19:39:11.482699 4313 tablet.cc:2369] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:11.513618 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.505859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:45101 (local address 127.0.0.1:44980)
0926 19:39:11.506133 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:11.506157 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:11.506322 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:11.506976 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:11.506981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:11.506996 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:39:11.507206 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.507218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.508820 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.508826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.510597 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.510614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.511643 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.511650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.511864 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.511947 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:39:11.513051 (+ 1104us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:39:11.513063 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 19:39:11.513133 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 19:39:11.513310 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 19:39:11.514369 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.506320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.2.164.65:45101)
0926 19:39:11.506513 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.506519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.506538 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:39:11.506648 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.506653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.506756 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.506907 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:11.507389 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.508595 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.510763 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.511370 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.511467 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.512168 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:11.512826 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:11.512950 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.513992 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.514082 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 19:39:11.519797 4262 raft_consensus.cc:1261] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 1 FOLLOWER]: Refusing update from remote peer f04c989a08d2423aae06476aea8cc63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:11.520082 4123 raft_consensus.cc:1261] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0 [term 1 FOLLOWER]: Refusing update from remote peer f04c989a08d2423aae06476aea8cc63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:11.520648 4340 consensus_queue.cc:1035] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:11.520977 4363 consensus_queue.cc:1035] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:11.546046 4337 tablet_replica.cc:439] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2: TxnStatusTablet state changed. Reason: New leader f04c989a08d2423aae06476aea8cc63a. Latest consensus state: current_term: 1 leader_uuid: "f04c989a08d2423aae06476aea8cc63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } }
I20260926 19:39:11.546434 4337 tablet_replica.cc:442] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:11.548238 4340 tablet_replica.cc:439] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f04c989a08d2423aae06476aea8cc63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } }
I20260926 19:39:11.548543 4340 tablet_replica.cc:442] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:39:11.550056 4344 mvcc.cc:205] Tried to move back new op lower bound from 7333689555013267456 to 7333689554845298688. Current Snapshot: MvccSnapshot[applied={T|T < 7333689555013267456}]
I20260926 19:39:11.550072 4339 tablet_replica.cc:439] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0: TxnStatusTablet state changed. Reason: New leader f04c989a08d2423aae06476aea8cc63a. Latest consensus state: current_term: 1 leader_uuid: "f04c989a08d2423aae06476aea8cc63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } }
I20260926 19:39:11.550312 4339 tablet_replica.cc:442] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:11.552671 4337 tablet_replica.cc:439] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f04c989a08d2423aae06476aea8cc63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } }
I20260926 19:39:11.552898 4337 tablet_replica.cc:442] T b27612364cb74ac5b9b8386a9dd3d5f3 P 9ebb92e730eb42b8b96d4f1f7afd99f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:11.553869 4363 tablet_replica.cc:439] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f04c989a08d2423aae06476aea8cc63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } }
I20260926 19:39:11.554100 4363 tablet_replica.cc:442] T b27612364cb74ac5b9b8386a9dd3d5f3 P f04c989a08d2423aae06476aea8cc63a: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:39:11.555213 4345 mvcc.cc:205] Tried to move back new op lower bound from 7333689555013267456 to 7333689554845298688. Current Snapshot: MvccSnapshot[applied={T|T < 7333689555013267456}]
I20260926 19:39:11.557840 4339 tablet_replica.cc:439] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f04c989a08d2423aae06476aea8cc63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04c989a08d2423aae06476aea8cc63a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45101 } } peers { permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 } } peers { permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 } } }
I20260926 19:39:11.558041 4339 tablet_replica.cc:442] T b27612364cb74ac5b9b8386a9dd3d5f3 P acc1b16893204cc4a2bd18d64d830de0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:11.564364 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.558270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.2.164.65:45101)
0926 19:39:11.558435 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.558440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.558456 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:39:11.558510 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.558515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.558595 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.558690 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:11.559267 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.560421 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.562343 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.563047 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.563182 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.563653 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:11.563777 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:11.563878 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.564012 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.564088 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:39:11.583016 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.575694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:37703 (local address 127.2.164.65:45101)
0926 19:39:11.575863 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.575868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.576495 (+ 627us) server_negotiation.cc:407] Connection header received
0926 19:39:11.576770 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.576780 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.576860 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.576943 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:11.577626 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.578725 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.580822 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.581494 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.581614 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.582144 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:11.582286 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:11.582372 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.582582 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.582692 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":57}
I20260926 19:39:11.583158 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.575743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:45101 (local address 127.2.164.65:37703)
0926 19:39:11.576386 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:11.576405 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:11.576556 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:11.576991 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:11.576996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:11.577009 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:39:11.577443 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.577455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.578856 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.578861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.580680 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:11.580694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.581690 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.581697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:11.581881 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.581966 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:39:11.582441 (+ 475us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:39:11.582453 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 19:39:11.582550 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 19:39:11.582719 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":22,"thread_start_us":334,"threads_started":1}
I20260926 19:39:11.594755 4123 raft_consensus.cc:1261] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0 [term 1 FOLLOWER]: Refusing update from remote peer f04c989a08d2423aae06476aea8cc63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:11.594779 4261 raft_consensus.cc:1261] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 1 FOLLOWER]: Refusing update from remote peer f04c989a08d2423aae06476aea8cc63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:11.595309 4340 consensus_queue.cc:1035] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb92e730eb42b8b96d4f1f7afd99f2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:11.595757 4340 consensus_queue.cc:1035] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc1b16893204cc4a2bd18d64d830de0" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:11.685420 3828 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:11.685619 3828 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:11.688230 3828 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 705046327de74bea86ef62d92aa1f527: Sending DeleteTablet for 3 replicas of tablet aa6df1b2b65d48cb9ef91247aad4c4c8
I20260926 19:39:11.689005 3964 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6df1b2b65d48cb9ef91247aad4c4c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:11 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:44964
I20260926 19:39:11.689103 4103 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6df1b2b65d48cb9ef91247aad4c4c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:11 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:35758
I20260926 19:39:11.689294 4355 tablet_replica.cc:330] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a: stopping tablet replica
I20260926 19:39:11.689378 4356 tablet_replica.cc:330] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0: stopping tablet replica
I20260926 19:39:11.689445 4355 raft_consensus.cc:2229] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:39:11.689594 4356 raft_consensus.cc:2229] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:11.689769 4356 raft_consensus.cc:2258] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:11.689841 4355 raft_consensus.cc:2258] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:11.690675 4242 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6df1b2b65d48cb9ef91247aad4c4c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:11 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:51838
I20260926 19:39:11.690968 4357 tablet_replica.cc:330] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2: stopping tablet replica
I20260926 19:39:11.691143 4355 ts_tablet_manager.cc:1905] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:11.691169 4357 raft_consensus.cc:2229] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:11.691324 4357 raft_consensus.cc:2258] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:11.692538 4357 ts_tablet_manager.cc:1905] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:11.699254 4356 ts_tablet_manager.cc:1905] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:11.699263 4357 ts_tablet_manager.cc:1918] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:39:11.699404 4357 log.cc:1196] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/aa6df1b2b65d48cb9ef91247aad4c4c8
I20260926 19:39:11.699769 4357 ts_tablet_manager.cc:1939] T aa6df1b2b65d48cb9ef91247aad4c4c8 P 9ebb92e730eb42b8b96d4f1f7afd99f2: Deleting consensus metadata
I20260926 19:39:11.700785 3813 catalog_manager.cc:4841] TS 9ebb92e730eb42b8b96d4f1f7afd99f2 (127.2.164.67:45717): tablet aa6df1b2b65d48cb9ef91247aad4c4c8 (table test-table [id=3190d144bd86437887e08c3ea722d061]) successfully deleted
I20260926 19:39:11.704212 4355 ts_tablet_manager.cc:1918] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:39:11.704357 4355 log.cc:1196] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/aa6df1b2b65d48cb9ef91247aad4c4c8
I20260926 19:39:11.704730 4355 ts_tablet_manager.cc:1939] T aa6df1b2b65d48cb9ef91247aad4c4c8 P f04c989a08d2423aae06476aea8cc63a: Deleting consensus metadata
I20260926 19:39:11.705845 3813 catalog_manager.cc:4841] TS f04c989a08d2423aae06476aea8cc63a (127.2.164.65:45101): tablet aa6df1b2b65d48cb9ef91247aad4c4c8 (table test-table [id=3190d144bd86437887e08c3ea722d061]) successfully deleted
I20260926 19:39:11.706610 4356 ts_tablet_manager.cc:1918] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:39:11.706811 4356 log.cc:1196] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/aa6df1b2b65d48cb9ef91247aad4c4c8
I20260926 19:39:11.707250 4356 ts_tablet_manager.cc:1939] T aa6df1b2b65d48cb9ef91247aad4c4c8 P acc1b16893204cc4a2bd18d64d830de0: Deleting consensus metadata
I20260926 19:39:11.708276 3812 catalog_manager.cc:4841] TS acc1b16893204cc4a2bd18d64d830de0 (127.2.164.66:37551): tablet aa6df1b2b65d48cb9ef91247aad4c4c8 (table test-table [id=3190d144bd86437887e08c3ea722d061]) successfully deleted
Sep 26 19:39:11 dist-test-slave-mrw4 krb5kdc[3783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:11.724341 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.710198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.2.164.65:45101)
0926 19:39:11.710360 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.710368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.710935 (+ 567us) server_negotiation.cc:407] Connection header received
0926 19:39:11.711096 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.711101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.711179 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.711254 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:11.712645 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.713437 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.714687 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.715106 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.715208 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.718988 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:11.719000 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:11.719015 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:11.719050 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:11.721901 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.722486 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.722493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.722495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.722611 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.722940 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.722944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.722946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.723289 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:11.723577 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.723826 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.724051 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 19:39:11.725092 4029 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44998
I20260926 19:39:11.742486 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.729153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.2.164.126:34743)
0926 19:39:11.729444 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.729450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.730073 (+ 623us) server_negotiation.cc:407] Connection header received
0926 19:39:11.730189 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.730194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.730275 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.730378 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:11.731456 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.732559 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.733790 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.734482 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.734624 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.737296 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:11.737302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:11.737307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:11.737339 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:11.739670 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.740468 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.740474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.740477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.740604 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.740993 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.740998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.741001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.741328 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:11.741559 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.741932 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.742159 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 19:39:11.743016 3828 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44038
I20260926 19:39:11.746074 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 3899
I20260926 19:39:11.760848 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.746765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.2.164.65:45101)
0926 19:39:11.746935 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.746941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.747214 (+ 273us) server_negotiation.cc:407] Connection header received
0926 19:39:11.747323 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.747328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.747397 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.747504 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:11.748465 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.749625 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.751228 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.751908 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.751997 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.754761 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:11.754768 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:11.754772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:11.754812 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:11.758144 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.758829 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.758834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.758838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.758996 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.759361 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.759366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.759369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.759694 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:11.759971 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.760209 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.760479 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 19:39:11.761404 4029 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45012
W20260926 19:39:11.761960 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 3899: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:39:11.762118 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 3899
I20260926 19:39:11.777103 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 4038
Sep 26 19:39:11 dist-test-slave-mrw4 krb5kdc[3783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:39:11.789834 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.777858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.2.164.66:37551)
0926 19:39:11.778023 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.778028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.778044 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:39:11.778094 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.778097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.778155 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.778230 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:11.779247 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.780030 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.781205 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.781612 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.781708 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.785102 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:11.785114 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:11.785128 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:11.785157 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:11.787569 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.788111 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.788118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.788120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.788242 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.788517 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.788520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.788521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.788825 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:11.789038 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.789353 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.789530 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 19:39:11.790431 4167 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35762
W20260926 19:39:11.790948 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 4038: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:39:11.791070 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 4038
I20260926 19:39:11.803826 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 4177
Sep 26 19:39:11 dist-test-slave-mrw4 krb5kdc[3783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:39:11.817234 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:11.804361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.2.164.67:45717)
0926 19:39:11.804753 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:11.804758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:11.804771 (+ 13us) server_negotiation.cc:407] Connection header received
0926 19:39:11.804814 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:11.804818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:11.804871 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:11.804977 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:11.806093 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.806758 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.807884 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:11.808284 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:11.808385 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:11.812151 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:11.812164 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:11.812179 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:11.812212 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:11.814767 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.815338 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.815345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.815347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.815483 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:11.815803 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:11.815807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:11.815809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:11.816170 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:11.816454 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:11.816704 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 19:39:11.816918 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":183,"threads_started":1}
W20260926 19:39:11.817883 4307 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51850
W20260926 19:39:11.818477 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 4177: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:39:11.818594 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 4177
I20260926 19:39:11.834573 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 3796
I20260926 19:39:11.835569 3868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:11.947661 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 3796
2026-09-26T19:39:11Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4001 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:12 dist-test-slave-mrw4 krb5kdc[4399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:12 dist-test-slave-mrw4 krb5kdc[4399](info): set up 2 sockets
Sep 26 19:39:12 dist-test-slave-mrw4 krb5kdc[4399](info): commencing operation
krb5kdc: starting...
W20260926 19:39:12.945094 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:13 dist-test-slave-mrw4 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:39:13.102595 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:42851
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:42851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:13.397771 4412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:13.398018 4412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:13.398109 4412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:13.405087 4412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:13.405196 4412 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 19:39:13.405248 4412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:13.405309 4412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:13.405355 4412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:13.405411 4412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:13.414755 4412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:42851
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:42851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:13.415900 4412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:13.417071 4412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:13.427716 4420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:13.427788 4418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:13.427788 4417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:13.428347 4412 server_base.cc:1034] running on GCE node
I20260926 19:39:13.429009 4412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:13.430286 4412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:13.431507 4412 hybrid_clock.cc:648] HybridClock initialized: now 1790451553431451 us; error 37 us; skew 500 ppm
I20260926 19:39:13.431811 4412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:13 dist-test-slave-mrw4 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:13.434666 4412 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:39:13.435946 4412 webserver.cc:460] Webserver started at http://127.2.164.126:42775/ using document root <none> and password file <none>
I20260926 19:39:13.436571 4412 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:13.436661 4412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:13.436937 4412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:13.439653 4412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "6b5f64af1c364873b5814dccc66161a4"
format_stamp: "Formatted at 2026-09-26 19:39:13 on dist-test-slave-mrw4"
server_key: "c5c650ba51e7e6f9887cf4dd5c9ad83a"
server_key_iv: "2f1b70ad2eba629ae4288f110c1473a4"
server_key_version: "encryptionkey@0"
I20260926 19:39:13.440337 4412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "6b5f64af1c364873b5814dccc66161a4"
format_stamp: "Formatted at 2026-09-26 19:39:13 on dist-test-slave-mrw4"
server_key: "c5c650ba51e7e6f9887cf4dd5c9ad83a"
server_key_iv: "2f1b70ad2eba629ae4288f110c1473a4"
server_key_version: "encryptionkey@0"
I20260926 19:39:13.444926 4412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 19:39:13.448076 4427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:13.449321 4412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:39:13.449448 4412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "6b5f64af1c364873b5814dccc66161a4"
format_stamp: "Formatted at 2026-09-26 19:39:13 on dist-test-slave-mrw4"
server_key: "c5c650ba51e7e6f9887cf4dd5c9ad83a"
server_key_iv: "2f1b70ad2eba629ae4288f110c1473a4"
server_key_version: "encryptionkey@0"
I20260926 19:39:13.449605 4412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:13.465314 4412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:13.469026 4412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:13.469274 4412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:13.487541 4412 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:42851
I20260926 19:39:13.487561 4488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:42851 every 8 connection(s)
I20260926 19:39:13.488751 4412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:13.492270 4489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:13.495474 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 4412
I20260926 19:39:13.495679 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:13.496434 2705 external_mini_cluster.cc:1428] Setting key efec7a907bcdccd3a256def776b0f210
I20260926 19:39:13.504429 4489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4: Bootstrap starting.
I20260926 19:39:13.508008 4489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:13.509219 4489 log.cc:824] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:39:13 dist-test-slave-mrw4 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:13.511917 4489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4: No bootstrap required, opened a new log
I20260926 19:39:13.518040 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:13.498324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.2.164.126:42851)
0926 19:39:13.499007 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:13.499018 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:13.499056 (+ 38us) server_negotiation.cc:407] Connection header received
0926 19:39:13.500624 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:13.500645 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:13.501034 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:13.501408 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:13.503036 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:13.504277 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:13.505972 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:13.506647 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:13.506780 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:13.511310 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:13.511317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:13.511331 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:13.511368 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:13.514281 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:13.514905 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:13.514916 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:13.514922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:13.515096 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:13.515440 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:13.515443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:13.515445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:13.515954 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:13.516245 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:13.516658 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 19:39:13.517044 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":305,"threads_started":1}
I20260926 19:39:13.519358 4489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f64af1c364873b5814dccc66161a4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42851 } }
I20260926 19:39:13.519770 4489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:13.519872 4489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b5f64af1c364873b5814dccc66161a4, State: Initialized, Role: FOLLOWER
I20260926 19:39:13.520485 4489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f64af1c364873b5814dccc66161a4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42851 } }
I20260926 19:39:13.520740 4489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:13.520871 4489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:13.520991 4489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:13.523319 4489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f64af1c364873b5814dccc66161a4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42851 } }
I20260926 19:39:13.523722 4489 leader_election.cc:302] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b5f64af1c364873b5814dccc66161a4; no voters:
I20260926 19:39:13.524166 4489 leader_election.cc:288] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:13.524289 4494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:13.524919 4494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [term 1 LEADER]: Becoming Leader. State: Replica: 6b5f64af1c364873b5814dccc66161a4, State: Running, Role: LEADER
I20260926 19:39:13.525410 4494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f64af1c364873b5814dccc66161a4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42851 } }
I20260926 19:39:13.525960 4489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:13.528060 4496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b5f64af1c364873b5814dccc66161a4. Latest consensus state: current_term: 1 leader_uuid: "6b5f64af1c364873b5814dccc66161a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f64af1c364873b5814dccc66161a4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42851 } } }
I20260926 19:39:13.528451 4496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:13.528035 4495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b5f64af1c364873b5814dccc66161a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f64af1c364873b5814dccc66161a4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 42851 } } }
I20260926 19:39:13.529352 4495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:13.530493 4502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:13.534420 4502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:13.542441 4502 catalog_manager.cc:1347] Generated new cluster ID: 411d56c462ba47fe81e94b9445fc2ae5
I20260926 19:39:13.542541 4502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:13.561277 4502 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:13.562918 4502 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:13.573957 4502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4: Generated new TSK 0
I20260926 19:39:13.574918 4502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:39:13.617990 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42851
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:13.915869 4515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:13.916141 4515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:13.916267 4515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:13.923195 4515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:13.923314 4515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:13.923434 4515 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 19:39:13.923496 4515 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:39:13.923599 4515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:13.932957 4515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:42851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:13.934242 4515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:13.935523 4515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:13.947258 4523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:13.947252 4521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:13.947474 4520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:13.948025 4515 server_base.cc:1034] running on GCE node
I20260926 19:39:13.948542 4515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:13.949432 4515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:13.950625 4515 hybrid_clock.cc:648] HybridClock initialized: now 1790451553950578 us; error 39 us; skew 500 ppm
I20260926 19:39:13.950919 4515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:13 dist-test-slave-mrw4 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:13.953791 4515 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:39:13.955116 4515 webserver.cc:460] Webserver started at http://127.2.164.65:39235/ using document root <none> and password file <none>
I20260926 19:39:13.955718 4515 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:13.955825 4515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:13.956089 4515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:13.958760 4515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a"
format_stamp: "Formatted at 2026-09-26 19:39:13 on dist-test-slave-mrw4"
server_key: "b8409c0e49e9c6eb0e3acc2aca7357eb"
server_key_iv: "8a6cef25e545fd645023e39e6dad0da8"
server_key_version: "encryptionkey@0"
I20260926 19:39:13.959442 4515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a"
format_stamp: "Formatted at 2026-09-26 19:39:13 on dist-test-slave-mrw4"
server_key: "b8409c0e49e9c6eb0e3acc2aca7357eb"
server_key_iv: "8a6cef25e545fd645023e39e6dad0da8"
server_key_version: "encryptionkey@0"
I20260926 19:39:13.963970 4515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:39:13.967077 4530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:13.968266 4515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:13.968418 4515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a"
format_stamp: "Formatted at 2026-09-26 19:39:13 on dist-test-slave-mrw4"
server_key: "b8409c0e49e9c6eb0e3acc2aca7357eb"
server_key_iv: "8a6cef25e545fd645023e39e6dad0da8"
server_key_version: "encryptionkey@0"
I20260926 19:39:13.968580 4515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:13.998353 4515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:14.002337 4515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:14.002597 4515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:14.005218 4515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:14.005329 4515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:14.005437 4515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:14.005519 4515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:39:14 dist-test-slave-mrw4 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:14.028012 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.005972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:52873 (local address 127.2.164.126:42851)
0926 19:39:14.006153 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:14.006158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:14.007482 (+ 1324us) server_negotiation.cc:407] Connection header received
0926 19:39:14.009565 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:14.009571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:14.009654 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:14.009752 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:14.012199 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.013106 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.015921 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.016454 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.016584 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:14.021320 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:14.021328 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:14.021332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:14.021375 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:14.024562 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:14.025464 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:14.025469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:14.025473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:14.025605 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:14.026203 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:14.026209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:14.026213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:14.026609 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:14.026754 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:14.027418 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 19:39:14.027642 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:39:14.028884 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.006561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42851 (local address 127.2.164.65:52873)
0926 19:39:14.007324 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:14.007363 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:14.009300 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:14.010206 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:14.010221 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:14.011051 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:14.011995 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:14.012025 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.013328 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.013334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:14.015759 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:14.015779 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.016680 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.016691 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:14.016973 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:14.017821 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:14.017858 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:14.021048 (+ 3190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:14.024707 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:14.024719 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:14.024739 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:14.025321 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:14.025716 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:14.025722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:14.025727 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:14.026024 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:14.026730 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:14.026735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:14.027057 (+ 322us) client_negotiation.cc:769] Sending connection context
0926 19:39:14.027326 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 19:39:14.027720 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":321,"threads_started":1}
I20260926 19:39:14.048867 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.036495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42851 (local address 127.2.164.65:43147)
0926 19:39:14.037192 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:14.037211 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:14.037345 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:14.037665 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:14.037669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:14.038065 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:14.038331 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:14.038339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.039332 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.039338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:14.040316 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:14.040327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.040990 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.040996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:14.041143 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:14.041850 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:14.041868 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:14.042907 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:14.046368 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:14.046373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:14.046377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:14.046972 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:14.047374 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:14.047377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:14.047379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:14.047490 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:14.048038 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:14.048041 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:14.048198 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 19:39:14.048304 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 19:39:14.048535 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":364,"threads_started":1}
I20260926 19:39:14.048974 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.036445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:43147 (local address 127.2.164.126:42851)
0926 19:39:14.036658 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:14.036665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:14.037291 (+ 626us) server_negotiation.cc:407] Connection header received
0926 19:39:14.037457 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:14.037463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:14.037532 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:14.037610 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:14.038458 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.039209 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.040440 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.040827 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.040930 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:14.043139 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:14.043143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:14.043147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:14.043176 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:14.046245 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:14.047098 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:14.047104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:14.047107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:14.047261 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:14.047573 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:14.047577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:14.047581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:14.047922 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:14.048066 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:14.048403 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 19:39:14.048636 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:39:14.056372 4515 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:40883
I20260926 19:39:14.056416 4649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:40883 every 8 connection(s)
I20260926 19:39:14.057538 4515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:14.059234 4650 heartbeater.cc:347] Connected to a master server at 127.2.164.126:42851
I20260926 19:39:14.059408 4650 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:14.059937 4650 heartbeater.cc:511] Master 127.2.164.126:42851 requested a full tablet report, sending...
I20260926 19:39:14.061751 4444 ts_manager.cc:194] Registered new tserver with Master: 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883)
I20260926 19:39:14.063112 4444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:52873
I20260926 19:39:14.063503 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 4515
I20260926 19:39:14.063683 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:14.064307 2705 external_mini_cluster.cc:1428] Setting key 926ab62463c3ecc12410e600e0597dc1
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:39:14.098614 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42851
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:14.400127 4654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:14.400381 4654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:14.400487 4654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:14.407600 4654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:14.407701 4654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:14.407789 4654 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 19:39:14.407837 4654 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:39:14.407902 4654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:14.417016 4654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:42851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:14.418164 4654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:14.419342 4654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:14.431068 4659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:14.431197 4662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:14.431077 4660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:14.431767 4654 server_base.cc:1034] running on GCE node
I20260926 19:39:14.432345 4654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:14.433210 4654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:14.434412 4654 hybrid_clock.cc:648] HybridClock initialized: now 1790451554434367 us; error 37 us; skew 500 ppm
I20260926 19:39:14.434708 4654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:14 dist-test-slave-mrw4 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451554, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:14.437503 4654 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:39:14.438874 4654 webserver.cc:460] Webserver started at http://127.2.164.66:34583/ using document root <none> and password file <none>
I20260926 19:39:14.439471 4654 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:14.439563 4654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:14.439808 4654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:14.442499 4654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "ba204f5b95274c3ea410a5f2c4da0c1d"
format_stamp: "Formatted at 2026-09-26 19:39:14 on dist-test-slave-mrw4"
server_key: "ed452350433e2c0eb0f07b71aa840644"
server_key_iv: "0c4309b577d242d14eb9d5e0f07b449b"
server_key_version: "encryptionkey@0"
I20260926 19:39:14.443156 4654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "ba204f5b95274c3ea410a5f2c4da0c1d"
format_stamp: "Formatted at 2026-09-26 19:39:14 on dist-test-slave-mrw4"
server_key: "ed452350433e2c0eb0f07b71aa840644"
server_key_iv: "0c4309b577d242d14eb9d5e0f07b449b"
server_key_version: "encryptionkey@0"
I20260926 19:39:14.447955 4654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 19:39:14.451251 4669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:14.452562 4654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:39:14.452716 4654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "ba204f5b95274c3ea410a5f2c4da0c1d"
format_stamp: "Formatted at 2026-09-26 19:39:14 on dist-test-slave-mrw4"
server_key: "ed452350433e2c0eb0f07b71aa840644"
server_key_iv: "0c4309b577d242d14eb9d5e0f07b449b"
server_key_version: "encryptionkey@0"
I20260926 19:39:14.452880 4654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:14.478497 4654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:14.482249 4654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:14.482481 4654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:14.484874 4654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:14.484957 4654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:14.485109 4654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:14.485234 4654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:39:14 dist-test-slave-mrw4 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451554, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:14.507053 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.485252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:38447 (local address 127.2.164.126:42851)
0926 19:39:14.485427 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:14.485433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:14.486806 (+ 1373us) server_negotiation.cc:407] Connection header received
0926 19:39:14.488806 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:14.488811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:14.488892 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:14.488984 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:14.491254 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.492350 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.494321 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.494943 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.495066 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:14.498982 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:14.498988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:14.498993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:14.499034 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:14.502845 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:14.503892 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:14.503898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:14.503902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:14.504051 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:14.504630 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:14.504636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:14.504640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:14.505024 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:14.505229 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:14.506518 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0926 19:39:14.506760 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:39:14.508270 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.485826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42851 (local address 127.2.164.66:38447)
0926 19:39:14.486640 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:14.486686 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:14.488514 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:14.489405 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:14.489415 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:14.490158 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:14.491029 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:14.491060 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.492568 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.492574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:14.494189 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:14.494200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.495288 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.495293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:14.495463 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:14.496065 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:14.496094 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:14.498769 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:14.503032 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:14.503045 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:14.503062 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:14.503736 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:14.504210 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:14.504214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:14.504218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:14.504478 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:14.505297 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:14.505305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:14.505837 (+ 532us) client_negotiation.cc:769] Sending connection context
0926 19:39:14.506173 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 19:39:14.506747 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":332,"threads_started":1}
I20260926 19:39:14.531140 4744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.514656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42851 (local address 127.2.164.66:35705)
0926 19:39:14.517918 (+ 3262us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:14.517939 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:14.518107 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:14.518414 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:14.518420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:14.519007 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:14.519388 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:14.519401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.520378 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.520384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:14.522187 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:14.522206 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.523016 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.523022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:14.523215 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:14.523879 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:14.523909 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:14.525136 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:14.528368 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:14.528376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:14.528381 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:14.528920 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:14.529397 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:14.529401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:14.529404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:14.529560 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:14.530251 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:14.530254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:14.530399 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 19:39:14.530497 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 19:39:14.530774 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3105,"thread_start_us":2967,"threads_started":1}
I20260926 19:39:14.531257 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.514641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:35705 (local address 127.2.164.126:42851)
0926 19:39:14.514835 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:14.514842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:14.518030 (+ 3188us) server_negotiation.cc:407] Connection header received
0926 19:39:14.518234 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:14.518238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:14.518297 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:14.518367 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:14.519507 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.520202 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.522328 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.522819 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.522937 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:14.525344 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:14.525350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:14.525354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:14.525390 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:14.528250 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:14.529083 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:14.529088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:14.529092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:14.529263 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:14.529695 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:14.529701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:14.529705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:14.530142 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:14.530297 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:14.530617 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 19:39:14.530854 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:39:14.540659 4654 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:32961
I20260926 19:39:14.540709 4788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:32961 every 8 connection(s)
I20260926 19:39:14.541816 4654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:14.542963 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 4654
I20260926 19:39:14.543130 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:14.543671 2705 external_mini_cluster.cc:1428] Setting key c76f097a691406249ada515b80ae2c6e
I20260926 19:39:14.543668 4789 heartbeater.cc:347] Connected to a master server at 127.2.164.126:42851
I20260926 19:39:14.543862 4789 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:14.544484 4789 heartbeater.cc:511] Master 127.2.164.126:42851 requested a full tablet report, sending...
I20260926 19:39:14.545708 4444 ts_manager.cc:194] Registered new tserver with Master: ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66:32961)
I20260926 19:39:14.546737 4444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:38447
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:39:14.577945 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:42851
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:14.870620 4793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:14.870857 4793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:14.870975 4793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:14.877846 4793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:14.877936 4793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:14.878050 4793 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 19:39:14.878108 4793 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:39:14.878201 4793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:14.887187 4793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:42851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:14.888319 4793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:14.889504 4793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:14.901098 4801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:14.901146 4799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:14.901328 4798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:14.901928 4793 server_base.cc:1034] running on GCE node
I20260926 19:39:14.902459 4793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:14.903285 4793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:14.904469 4793 hybrid_clock.cc:648] HybridClock initialized: now 1790451554904416 us; error 49 us; skew 500 ppm
I20260926 19:39:14.904729 4793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:14 dist-test-slave-mrw4 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451554, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:14.907454 4793 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:39:14.908730 4793 webserver.cc:460] Webserver started at http://127.2.164.67:39213/ using document root <none> and password file <none>
I20260926 19:39:14.909351 4793 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:14.909457 4793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:14.909713 4793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:14.912259 4793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "7aa94f8ca8f2478a9215341b8e12a842"
format_stamp: "Formatted at 2026-09-26 19:39:14 on dist-test-slave-mrw4"
server_key: "9f41991d940e52ee75767962d23f43cd"
server_key_iv: "393e88d45f03360cf1e85c6a63599844"
server_key_version: "encryptionkey@0"
I20260926 19:39:14.912914 4793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "7aa94f8ca8f2478a9215341b8e12a842"
format_stamp: "Formatted at 2026-09-26 19:39:14 on dist-test-slave-mrw4"
server_key: "9f41991d940e52ee75767962d23f43cd"
server_key_iv: "393e88d45f03360cf1e85c6a63599844"
server_key_version: "encryptionkey@0"
I20260926 19:39:14.917424 4793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:39:14.920488 4808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:14.921726 4793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:39:14.921856 4793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "7aa94f8ca8f2478a9215341b8e12a842"
format_stamp: "Formatted at 2026-09-26 19:39:14 on dist-test-slave-mrw4"
server_key: "9f41991d940e52ee75767962d23f43cd"
server_key_iv: "393e88d45f03360cf1e85c6a63599844"
server_key_version: "encryptionkey@0"
I20260926 19:39:14.922009 4793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:14.963227 4793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:14.967096 4793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:14.967324 4793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:14.969727 4793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:14.969820 4793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:14.969924 4793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:14.969991 4793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:39:14 dist-test-slave-mrw4 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451554, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:14.991293 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.970337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:56267 (local address 127.2.164.126:42851)
0926 19:39:14.970517 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:14.970523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:14.971874 (+ 1351us) server_negotiation.cc:407] Connection header received
0926 19:39:14.974023 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:14.974029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:14.974112 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:14.974208 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:14.976360 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.977474 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.979736 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.980275 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.980391 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:14.984577 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:14.984582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:14.984588 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:14.984628 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:14.987615 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:14.988757 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:14.988762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:14.988766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:14.988928 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:14.989522 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:14.989527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:14.989530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:14.989890 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:14.990022 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:14.990715 (+ 693us) server_negotiation.cc:299] Negotiation successful
0926 19:39:14.990949 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:39:14.992311 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.970915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42851 (local address 127.2.164.67:56267)
0926 19:39:14.971720 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:14.971772 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:14.973676 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:14.974596 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:14.974607 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:14.975345 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:14.976141 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:14.976164 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.977652 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.977658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:14.979550 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:14.979569 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:14.980518 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:14.980526 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:14.980735 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:14.981351 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:14.981376 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:14.984303 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:14.987784 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:14.987793 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:14.987811 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:14.988583 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:14.989067 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:14.989075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:14.989078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:14.989400 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:14.990014 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:14.990026 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:14.990369 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 19:39:14.990597 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 19:39:14.991126 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":335,"threads_started":1}
I20260926 19:39:15.010612 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.997989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:37077 (local address 127.2.164.126:42851)
0926 19:39:14.998163 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:14.998169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:14.998808 (+ 639us) server_negotiation.cc:407] Connection header received
0926 19:39:14.998968 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:14.998972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:14.999026 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:14.999098 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:15.000171 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.000849 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.002892 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.003435 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.003549 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.005182 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:15.005188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:15.005192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:15.005226 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:15.008003 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:15.008586 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:15.008591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:15.008595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:15.008710 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:15.009014 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:15.009019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:15.009023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:15.009392 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:15.009602 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.010025 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.010244 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:39:15.010658 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:14.998071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42851 (local address 127.2.164.67:37077)
0926 19:39:14.998703 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:14.998724 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:14.998863 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:14.999136 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:14.999141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:14.999688 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:15.000043 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.000054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.000957 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.000963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.002750 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.002765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.003621 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.003627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.003785 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.004315 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:15.004331 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:15.005005 (+ 674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:15.008122 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:15.008128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:15.008132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:15.008484 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:15.008808 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:15.008812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:15.008816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:15.008923 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:15.009569 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:15.009574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:15.009773 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.009928 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.010229 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":321,"threads_started":1}
I20260926 19:39:15.018666 4793 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:43779
I20260926 19:39:15.018754 4927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:43779 every 8 connection(s)
I20260926 19:39:15.019804 4793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:15.021528 4928 heartbeater.cc:347] Connected to a master server at 127.2.164.126:42851
I20260926 19:39:15.021732 4928 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:15.021782 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 4793
I20260926 19:39:15.021943 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:15.022492 4928 heartbeater.cc:511] Master 127.2.164.126:42851 requested a full tablet report, sending...
I20260926 19:39:15.022513 2705 external_mini_cluster.cc:1428] Setting key b56bb337be2478c45f5c5348f81569e7
I20260926 19:39:15.023944 4443 ts_manager.cc:194] Registered new tserver with Master: 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779)
I20260926 19:39:15.024951 4443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:56267
I20260926 19:39:15.025357 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:39:15 dist-test-slave-mrw4 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 19:39:15 dist-test-slave-mrw4 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:15.065868 4650 heartbeater.cc:503] Master 127.2.164.126:42851 was elected leader, sending a full tablet report...
I20260926 19:39:15.067682 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.054582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55078 (local address 127.2.164.126:42851)
0926 19:39:15.054805 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.054812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.054976 (+ 164us) server_negotiation.cc:407] Connection header received
0926 19:39:15.055182 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.055187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.055274 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.055362 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:15.056831 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.057668 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.058864 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.059321 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.059435 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.062965 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:15.062970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:15.062973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:15.063009 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:15.065251 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:15.065806 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:15.065811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:15.065815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:15.065944 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:15.066337 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:15.066342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:15.066344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:15.066634 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:15.066854 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.067201 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.067415 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:39:15.071473 4443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:39:15.078012 4443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:15.100672 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.088994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43779 (local address 127.0.0.1:50332)
0926 19:39:15.089828 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.089881 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.090053 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.091040 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.091047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.091081 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:15.091514 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.091527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.095050 (+ 3523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.095057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.097520 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.097536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.099773 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.099779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.099991 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.100042 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.100169 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.100354 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":424,"threads_started":1}
I20260926 19:39:15.100881 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.089280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.2.164.67:43779)
0926 19:39:15.090401 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.090410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.090446 (+ 36us) server_negotiation.cc:407] Connection header received
0926 19:39:15.090542 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.090550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.090805 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.091719 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:15.093033 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.094851 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.098248 (+ 3397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.099594 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.099719 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.099950 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.100360 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.100526 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":772,"threads_started":1}
I20260926 19:39:15.104660 4858 tablet_service.cc:1467] Processing CreateTablet for tablet e40a4f25194742d99ecef5220e2e6dda (DEFAULT_TABLE table=test-table [id=dd83266f7b284d5bbed1082e42daa524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:15.105643 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.090826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32961 (local address 127.0.0.1:40416)
0926 19:39:15.097465 (+ 6639us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.097486 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.097591 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.099496 (+ 1905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.099501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.099527 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:15.099925 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.099935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.101898 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.101903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.103681 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.103692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.104856 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.104863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.105031 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.105056 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.105138 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.105293 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6514,"mutex_wait_us":5,"thread_start_us":6369,"threads_started":1}
I20260926 19:39:15.105645 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.091299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.2.164.66:32961)
0926 19:39:15.098918 (+ 7619us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.098926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.098954 (+ 28us) server_negotiation.cc:407] Connection header received
0926 19:39:15.099041 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.099048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.099306 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.099504 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:15.100114 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.101630 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.103823 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.104657 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.104794 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.104987 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.105220 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.105381 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7469,"thread_start_us":7346,"threads_started":1}
I20260926 19:39:15.106274 4858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40a4f25194742d99ecef5220e2e6dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:15.107903 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.090235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:40883 (local address 127.0.0.1:42974)
0926 19:39:15.090746 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.090769 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.090885 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.100894 (+ 10009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.100899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.100923 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:15.101264 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.101275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.103141 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.103147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.105099 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.105113 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.107210 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.107217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.107388 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.107420 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.107485 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.107618 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":255,"threads_started":1}
I20260926 19:39:15.108887 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.090550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.2.164.65:40883)
0926 19:39:15.100355 (+ 9805us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.100363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.100387 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:39:15.100472 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.100480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.100685 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.101439 (+ 754us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:15.101581 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.103000 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.105323 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.106982 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.108224 (+ 1242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.108418 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.108482 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.108627 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9659,"thread_start_us":9539,"threads_started":1}
I20260926 19:39:15.109030 4719 tablet_service.cc:1467] Processing CreateTablet for tablet e40a4f25194742d99ecef5220e2e6dda (DEFAULT_TABLE table=test-table [id=dd83266f7b284d5bbed1082e42daa524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:15.110338 4719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40a4f25194742d99ecef5220e2e6dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:15.112289 4580 tablet_service.cc:1467] Processing CreateTablet for tablet e40a4f25194742d99ecef5220e2e6dda (DEFAULT_TABLE table=test-table [id=dd83266f7b284d5bbed1082e42daa524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:15.113538 4580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40a4f25194742d99ecef5220e2e6dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:15.121194 4947 tablet_bootstrap.cc:492] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: Bootstrap starting.
I20260926 19:39:15.121872 4948 tablet_bootstrap.cc:492] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: Bootstrap starting.
I20260926 19:39:15.124475 4948 tablet_bootstrap.cc:654] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:15.125554 4948 log.cc:824] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:15.125599 4947 tablet_bootstrap.cc:654] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:15.126775 4950 tablet_bootstrap.cc:492] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a: Bootstrap starting.
I20260926 19:39:15.126776 4947 log.cc:824] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:15.128556 4948 tablet_bootstrap.cc:492] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: No bootstrap required, opened a new log
I20260926 19:39:15.128818 4948 ts_tablet_manager.cc:1397] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260926 19:39:15.130132 4950 tablet_bootstrap.cc:654] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:15.131181 4950 log.cc:824] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:15.134033 4947 tablet_bootstrap.cc:492] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: No bootstrap required, opened a new log
I20260926 19:39:15.134033 4950 tablet_bootstrap.cc:492] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a: No bootstrap required, opened a new log
I20260926 19:39:15.134289 4950 ts_tablet_manager.cc:1397] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 19:39:15.134305 4947 ts_tablet_manager.cc:1397] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260926 19:39:15.136703 4948 raft_consensus.cc:348] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } }
I20260926 19:39:15.137028 4948 raft_consensus.cc:374] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:15.137113 4948 raft_consensus.cc:729] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba204f5b95274c3ea410a5f2c4da0c1d, State: Initialized, Role: FOLLOWER
I20260926 19:39:15.137661 4948 consensus_queue.cc:260] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } }
I20260926 19:39:15.138581 4789 heartbeater.cc:503] Master 127.2.164.126:42851 was elected leader, sending a full tablet report...
I20260926 19:39:15.139186 4948 ts_tablet_manager.cc:1428] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260926 19:39:15.144042 4950 raft_consensus.cc:348] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } }
I20260926 19:39:15.144445 4950 raft_consensus.cc:374] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:15.144537 4950 raft_consensus.cc:729] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391ae9b2cdf347fcaecc27c6b7de7b5a, State: Initialized, Role: FOLLOWER
I20260926 19:39:15.145113 4950 consensus_queue.cc:260] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } }
I20260926 19:39:15.145306 4947 raft_consensus.cc:348] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } }
I20260926 19:39:15.145695 4947 raft_consensus.cc:374] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:15.145789 4947 raft_consensus.cc:729] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa94f8ca8f2478a9215341b8e12a842, State: Initialized, Role: FOLLOWER
I20260926 19:39:15.146332 4947 consensus_queue.cc:260] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } }
I20260926 19:39:15.146853 4950 ts_tablet_manager.cc:1428] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 19:39:15.147609 4928 heartbeater.cc:503] Master 127.2.164.126:42851 was elected leader, sending a full tablet report...
I20260926 19:39:15.147886 4947 ts_tablet_manager.cc:1428] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 19:39:15.199589 4955 raft_consensus.cc:482] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:15.199908 4955 raft_consensus.cc:504] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } }
I20260926 19:39:15.201591 4955 leader_election.cc:288] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883), ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66:32961)
I20260926 19:39:15.207790 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.202080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:54115 (local address 127.2.164.65:40883)
0926 19:39:15.202318 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.202324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.202341 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:39:15.202405 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.202409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.202474 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.202559 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:15.203072 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.204005 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.205824 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.206984 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.207095 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.207179 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.207507 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.207613 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 19:39:15.207859 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.201879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:40883 (local address 127.2.164.67:54115)
0926 19:39:15.202088 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.202106 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.202241 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.202707 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.202712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.202742 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:15.202912 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.202923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.204122 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.204128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.205654 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.205669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.207134 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.207140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.207342 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.207376 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.207468 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.207591 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":3}
I20260926 19:39:15.208875 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.202426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:55739 (local address 127.2.164.66:32961)
0926 19:39:15.202615 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.202620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.202969 (+ 349us) server_negotiation.cc:407] Connection header received
0926 19:39:15.203090 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.203093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.203151 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.203214 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:15.203789 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.204726 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.206941 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.208029 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.208179 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.208245 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.208519 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.208640 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:39:15.208926 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.202264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32961 (local address 127.2.164.67:55739)
0926 19:39:15.202863 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.202879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.202974 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.203254 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.203258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.203291 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:15.203611 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.203658 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.204875 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.204878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.206776 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.206791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.208233 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.208238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.208376 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.208398 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.208473 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.208583 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":19,"thread_start_us":291,"threads_started":1}
I20260926 19:39:15.208793 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40a4f25194742d99ecef5220e2e6dda" candidate_uuid: "7aa94f8ca8f2478a9215341b8e12a842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" is_pre_election: true
I20260926 19:39:15.209316 4600 raft_consensus.cc:2454] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aa94f8ca8f2478a9215341b8e12a842 in term 0.
I20260926 19:39:15.209792 4743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40a4f25194742d99ecef5220e2e6dda" candidate_uuid: "7aa94f8ca8f2478a9215341b8e12a842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" is_pre_election: true
I20260926 19:39:15.209995 4811 leader_election.cc:302] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391ae9b2cdf347fcaecc27c6b7de7b5a, 7aa94f8ca8f2478a9215341b8e12a842; no voters:
I20260926 19:39:15.210294 4743 raft_consensus.cc:2454] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aa94f8ca8f2478a9215341b8e12a842 in term 0.
I20260926 19:39:15.210419 4955 raft_consensus.cc:2781] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:15.210554 4955 raft_consensus.cc:482] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:15.210788 4955 raft_consensus.cc:3037] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:15.212949 4955 raft_consensus.cc:504] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } }
I20260926 19:39:15.213497 4955 leader_election.cc:288] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [CANDIDATE]: Term 1 election: Requested vote from peers 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883), ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66:32961)
I20260926 19:39:15.213891 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40a4f25194742d99ecef5220e2e6dda" candidate_uuid: "7aa94f8ca8f2478a9215341b8e12a842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a"
I20260926 19:39:15.213986 4743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40a4f25194742d99ecef5220e2e6dda" candidate_uuid: "7aa94f8ca8f2478a9215341b8e12a842" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d"
I20260926 19:39:15.214087 4600 raft_consensus.cc:3037] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:15.214175 4743 raft_consensus.cc:3037] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:15.217093 4600 raft_consensus.cc:2454] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aa94f8ca8f2478a9215341b8e12a842 in term 1.
I20260926 19:39:15.217095 4743 raft_consensus.cc:2454] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aa94f8ca8f2478a9215341b8e12a842 in term 1.
I20260926 19:39:15.217444 4811 leader_election.cc:302] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aa94f8ca8f2478a9215341b8e12a842, ba204f5b95274c3ea410a5f2c4da0c1d; no voters:
I20260926 19:39:15.217692 4955 raft_consensus.cc:2781] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:15.218073 4955 raft_consensus.cc:686] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 LEADER]: Becoming Leader. State: Replica: 7aa94f8ca8f2478a9215341b8e12a842, State: Running, Role: LEADER
I20260926 19:39:15.218477 4955 consensus_queue.cc:237] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } }
I20260926 19:39:15.222661 4444 catalog_manager.cc:5495] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "7aa94f8ca8f2478a9215341b8e12a842" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } health_report { overall_health: UNKNOWN } } }
Sep 26 19:39:15 dist-test-slave-mrw4 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
W20260926 19:39:15.271234 4929 tablet.cc:2369] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:15.275506 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.263346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:42851 (local address 127.0.0.1:55090)
0926 19:39:15.263990 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.264005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.264132 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.264419 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.264423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.265035 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:15.265343 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.265353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.266276 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.266280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.267250 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.267261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.267876 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.267880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.267994 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.268524 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:15.268543 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:15.270616 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:15.273406 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:15.273415 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:15.273421 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:15.273839 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:15.274128 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:15.274133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:15.274136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:15.274258 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:15.274690 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:15.274696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:15.274844 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.274985 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.275169 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":350,"threads_started":1}
I20260926 19:39:15.275705 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.263307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.2.164.126:42851)
0926 19:39:15.263494 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.263504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.264087 (+ 583us) server_negotiation.cc:407] Connection header received
0926 19:39:15.264221 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.264225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.264292 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.264369 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:15.265457 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.266168 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.267364 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.267756 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.267839 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.270785 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:15.270790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:15.270793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:15.270820 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:15.273306 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:15.273930 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:15.273933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:15.273936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:15.274046 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:15.274337 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:15.274340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:15.274342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:15.274558 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:15.274656 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.275060 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.275330 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":65}
I20260926 19:39:15.280135 4444 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:55090:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 19:39:15.281090 4444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:15.287400 4858 tablet_service.cc:1467] Processing CreateTablet for tablet bdc99ae099dd4930afc0281db383d6e1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50326e28956f4f998fc1ccb0bb0198a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:39:15.287321 4719 tablet_service.cc:1467] Processing CreateTablet for tablet bdc99ae099dd4930afc0281db383d6e1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50326e28956f4f998fc1ccb0bb0198a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:39:15.287248 4580 tablet_service.cc:1467] Processing CreateTablet for tablet bdc99ae099dd4930afc0281db383d6e1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50326e28956f4f998fc1ccb0bb0198a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:39:15.288125 4858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc99ae099dd4930afc0281db383d6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:15.288142 4719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc99ae099dd4930afc0281db383d6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:15.288153 4580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc99ae099dd4930afc0281db383d6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:15.292870 4947 tablet_bootstrap.cc:492] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842: Bootstrap starting.
W20260926 19:39:15.293586 4790 tablet.cc:2369] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:15.294416 4947 tablet_bootstrap.cc:654] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:15.294875 4950 tablet_bootstrap.cc:492] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Bootstrap starting.
I20260926 19:39:15.294895 4948 tablet_bootstrap.cc:492] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d: Bootstrap starting.
I20260926 19:39:15.296372 4950 tablet_bootstrap.cc:654] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:15.296411 4948 tablet_bootstrap.cc:654] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:15.297066 4947 tablet_bootstrap.cc:492] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842: No bootstrap required, opened a new log
I20260926 19:39:15.297240 4947 ts_tablet_manager.cc:1397] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 19:39:15.298027 4947 raft_consensus.cc:348] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.298257 4947 raft_consensus.cc:374] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:15.298380 4948 tablet_bootstrap.cc:492] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d: No bootstrap required, opened a new log
I20260926 19:39:15.298363 4947 raft_consensus.cc:729] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa94f8ca8f2478a9215341b8e12a842, State: Initialized, Role: FOLLOWER
I20260926 19:39:15.298501 4948 ts_tablet_manager.cc:1397] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260926 19:39:15.298626 4947 consensus_queue.cc:260] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.299036 4955 tablet_replica.cc:439] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } }
I20260926 19:39:15.299256 4955 tablet_replica.cc:442] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:15.299316 4948 raft_consensus.cc:348] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.299543 4948 raft_consensus.cc:374] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:15.299613 4950 tablet_bootstrap.cc:492] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: No bootstrap required, opened a new log
I20260926 19:39:15.299639 4948 raft_consensus.cc:729] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba204f5b95274c3ea410a5f2c4da0c1d, State: Initialized, Role: FOLLOWER
I20260926 19:39:15.299739 4950 ts_tablet_manager.cc:1397] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260926 19:39:15.299845 4948 consensus_queue.cc:260] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.300283 4947 ts_tablet_manager.cc:1428] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 19:39:15.300230 4953 tablet_replica.cc:439] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } }
I20260926 19:39:15.300566 4953 tablet_replica.cc:442] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:15.300436 4950 raft_consensus.cc:348] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.300640 4950 raft_consensus.cc:374] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:15.300719 4950 raft_consensus.cc:729] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391ae9b2cdf347fcaecc27c6b7de7b5a, State: Initialized, Role: FOLLOWER
I20260926 19:39:15.300935 4950 consensus_queue.cc:260] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.301102 4948 ts_tablet_manager.cc:1428] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 19:39:15.301520 4950 ts_tablet_manager.cc:1428] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 19:39:15.301376 4954 tablet_replica.cc:439] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } }
I20260926 19:39:15.301712 4954 tablet_replica.cc:442] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 19:39:15.309546 4651 tablet.cc:2369] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:15.519788 4954 raft_consensus.cc:482] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:15.520005 4954 raft_consensus.cc:504] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.521592 4954 leader_election.cc:288] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66:32961), 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779)
I20260926 19:39:15.528263 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.521874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32961 (local address 127.2.164.65:41107)
0926 19:39:15.522569 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.522588 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.522736 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.523123 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.523128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.523155 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:15.523517 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.523530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.524573 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.524578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.526480 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.526494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.527702 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.527706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.527835 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.527860 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.527930 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.528016 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":432,"threads_started":1}
I20260926 19:39:15.528376 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.521915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:41107 (local address 127.2.164.66:32961)
0926 19:39:15.522141 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.522149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.522695 (+ 546us) server_negotiation.cc:407] Connection header received
0926 19:39:15.522886 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.522892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.522970 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.523080 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:15.523693 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.524449 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.526634 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.527519 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.527674 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.527745 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.527969 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.528095 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 19:39:15.529222 4743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99ae099dd4930afc0281db383d6e1" candidate_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" is_pre_election: true
I20260926 19:39:15.529453 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.522688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43779 (local address 127.2.164.65:51855)
0926 19:39:15.523277 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.523290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.523387 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.524326 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.524329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.524348 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:15.524659 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.524673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.525786 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.525792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.527362 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.527373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.528739 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.528745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.528898 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.528921 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.528997 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.529117 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":291,"threads_started":1}
I20260926 19:39:15.529496 4743 raft_consensus.cc:2454] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 391ae9b2cdf347fcaecc27c6b7de7b5a in term 0.
I20260926 19:39:15.529678 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.522855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:51855 (local address 127.2.164.67:43779)
0926 19:39:15.523024 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.523029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.523371 (+ 342us) server_negotiation.cc:407] Connection header received
0926 19:39:15.523484 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.523489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.523559 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.523638 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:15.524905 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.525657 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.527496 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.528542 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.528701 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.528765 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.529040 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.529183 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:39:15.529974 4533 leader_election.cc:302] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391ae9b2cdf347fcaecc27c6b7de7b5a, ba204f5b95274c3ea410a5f2c4da0c1d; no voters:
I20260926 19:39:15.530375 4954 raft_consensus.cc:2781] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:15.530503 4954 raft_consensus.cc:482] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:15.530441 4878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99ae099dd4930afc0281db383d6e1" candidate_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa94f8ca8f2478a9215341b8e12a842" is_pre_election: true
I20260926 19:39:15.530596 4954 raft_consensus.cc:3037] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:15.530786 4878 raft_consensus.cc:2454] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 391ae9b2cdf347fcaecc27c6b7de7b5a in term 0.
I20260926 19:39:15.532801 4954 raft_consensus.cc:504] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.533373 4954 leader_election.cc:288] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [CANDIDATE]: Term 1 election: Requested vote from peers ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66:32961), 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779)
I20260926 19:39:15.533742 4743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99ae099dd4930afc0281db383d6e1" candidate_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d"
I20260926 19:39:15.533776 4878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc99ae099dd4930afc0281db383d6e1" candidate_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa94f8ca8f2478a9215341b8e12a842"
I20260926 19:39:15.533939 4878 raft_consensus.cc:3037] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:15.533959 4743 raft_consensus.cc:3037] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:15.536696 4878 raft_consensus.cc:2454] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 391ae9b2cdf347fcaecc27c6b7de7b5a in term 1.
I20260926 19:39:15.536684 4743 raft_consensus.cc:2454] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 391ae9b2cdf347fcaecc27c6b7de7b5a in term 1.
I20260926 19:39:15.537014 4533 leader_election.cc:302] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391ae9b2cdf347fcaecc27c6b7de7b5a, ba204f5b95274c3ea410a5f2c4da0c1d; no voters:
I20260926 19:39:15.537276 4954 raft_consensus.cc:2781] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:15.537662 4954 raft_consensus.cc:686] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 LEADER]: Becoming Leader. State: Replica: 391ae9b2cdf347fcaecc27c6b7de7b5a, State: Running, Role: LEADER
I20260926 19:39:15.538138 4954 consensus_queue.cc:237] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.541280 4970 tablet_replica.cc:439] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: TxnStatusTablet state changed. Reason: New leader 391ae9b2cdf347fcaecc27c6b7de7b5a. Latest consensus state: current_term: 1 leader_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } }
I20260926 19:39:15.541636 4970 tablet_replica.cc:442] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:39:15.542347 4972 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 19:39:15.542596 4972 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 19:39:15.542905 4443 catalog_manager.cc:5495] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65). New cstate: current_term: 1 leader_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } health_report { overall_health: UNKNOWN } } }
I20260926 19:39:15.600013 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.593998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.2.164.65:40883)
0926 19:39:15.594180 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.594187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.594335 (+ 148us) server_negotiation.cc:407] Connection header received
0926 19:39:15.594591 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.594595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.594685 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.594831 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:15.595266 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.596097 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.597942 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.598373 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.598474 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.598939 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.599434 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.599541 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.599686 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.599753 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:39:15.600069 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.593990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:40883 (local address 127.0.0.1:42984)
0926 19:39:15.594216 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.594234 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.594394 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.594863 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.594868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.594881 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:39:15.595093 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.595105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.596216 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.596222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.597813 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.597827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.598516 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.598522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.598715 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.598781 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.599535 (+ 754us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.599552 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.599631 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.599770 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 19:39:15.607645 4743 raft_consensus.cc:1261] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Refusing update from remote peer 391ae9b2cdf347fcaecc27c6b7de7b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:15.607667 4878 raft_consensus.cc:1261] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Refusing update from remote peer 391ae9b2cdf347fcaecc27c6b7de7b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:15.608579 4970 consensus_queue.cc:1035] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:15.608882 4954 consensus_queue.cc:1035] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:15.617138 4953 tablet_replica.cc:439] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d: TxnStatusTablet state changed. Reason: New leader 391ae9b2cdf347fcaecc27c6b7de7b5a. Latest consensus state: current_term: 1 leader_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } }
I20260926 19:39:15.617434 4953 tablet_replica.cc:442] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:15.620285 4970 tablet_replica.cc:439] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } }
I20260926 19:39:15.620735 4976 mvcc.cc:205] Tried to move back new op lower bound from 7333689571760410624 to 7333689571487707136. Current Snapshot: MvccSnapshot[applied={T|T < 7333689571760410624}]
I20260926 19:39:15.623890 4970 tablet_replica.cc:442] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:39:15.625149 4953 tablet_replica.cc:439] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } }
I20260926 19:39:15.625372 4953 tablet_replica.cc:442] T bdc99ae099dd4930afc0281db383d6e1 P ba204f5b95274c3ea410a5f2c4da0c1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:15.634753 4955 tablet_replica.cc:439] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842: TxnStatusTablet state changed. Reason: New leader 391ae9b2cdf347fcaecc27c6b7de7b5a. Latest consensus state: current_term: 1 leader_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } }
I20260926 19:39:15.635048 4955 tablet_replica.cc:442] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:15.636370 4970 tablet_replica.cc:439] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } }
I20260926 19:39:15.636628 4970 tablet_replica.cc:442] T bdc99ae099dd4930afc0281db383d6e1 P 391ae9b2cdf347fcaecc27c6b7de7b5a: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:39:15.644598 4955 tablet_replica.cc:439] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } } }
I20260926 19:39:15.644856 4955 tablet_replica.cc:442] T bdc99ae099dd4930afc0281db383d6e1 P 7aa94f8ca8f2478a9215341b8e12a842: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:39:15.655253 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.634650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.2.164.67:43779)
0926 19:39:15.634813 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.634819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.649732 (+ 14913us) server_negotiation.cc:407] Connection header received
0926 19:39:15.649931 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.649935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.650000 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.650112 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:15.650538 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.651263 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.653295 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.653692 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.653792 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.654240 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.654707 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.654823 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.654932 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.654992 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:39:15.669800 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.664397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:40883 (local address 127.2.164.67:59977)
0926 19:39:15.664919 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.664933 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.665040 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.665382 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.665385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.665393 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:39:15.665604 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.665610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.666998 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.667001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.667973 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.667983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.668854 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.668858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.668966 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.669011 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.669398 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.669406 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.669459 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.669548 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":287,"threads_started":1}
I20260926 19:39:15.669857 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.664381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:59977 (local address 127.2.164.65:40883)
0926 19:39:15.664515 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.664519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.665012 (+ 493us) server_negotiation.cc:407] Connection header received
0926 19:39:15.665184 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.665190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.665277 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.665370 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:15.665722 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.666877 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.668095 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.668710 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.668836 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.669198 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.669315 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.669400 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.669492 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.669567 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 19:39:15.678069 4743 raft_consensus.cc:1261] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Refusing update from remote peer 7aa94f8ca8f2478a9215341b8e12a842: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:15.678208 4600 raft_consensus.cc:1261] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Refusing update from remote peer 7aa94f8ca8f2478a9215341b8e12a842: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:15.678637 4957 consensus_queue.cc:1035] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:15.678957 4955 consensus_queue.cc:1035] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [LEADER]: Connected to new peer: Peer: permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:15.711539 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.705789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:41585 (local address 127.2.164.67:43779)
0926 19:39:15.705934 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.705939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.706472 (+ 533us) server_negotiation.cc:407] Connection header received
0926 19:39:15.706645 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.706650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.706722 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.706795 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:15.707485 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.708278 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.709767 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.710320 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.710447 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.710850 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.710961 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.711044 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.711189 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.711261 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 19:39:15.711773 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.705746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43779 (local address 127.2.164.65:41585)
0926 19:39:15.706370 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.706389 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.706515 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.706964 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.706969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.706981 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:39:15.707312 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.707325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.708433 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.708436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.709647 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.709658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.710487 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.710493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.710648 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.710700 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.711061 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.711073 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.711159 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.711288 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":323,"threads_started":1}
I20260926 19:39:15.744874 4443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:15.745127 4443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:15.748728 4443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4: Sending DeleteTablet for 3 replicas of tablet e40a4f25194742d99ecef5220e2e6dda
I20260926 19:39:15.749918 4580 tablet_service.cc:1514] Processing DeleteTablet for tablet e40a4f25194742d99ecef5220e2e6dda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:15 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:42974
I20260926 19:39:15.750875 4719 tablet_service.cc:1514] Processing DeleteTablet for tablet e40a4f25194742d99ecef5220e2e6dda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:15 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:40416
I20260926 19:39:15.750907 4443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:39:15.751478 4443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:15.753214 4858 tablet_service.cc:1514] Processing DeleteTablet for tablet e40a4f25194742d99ecef5220e2e6dda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:15 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:50332
I20260926 19:39:15.758209 4579 tablet_service.cc:1467] Processing CreateTablet for tablet d82946580e6f4408810d2d1af6bc69d5 (DEFAULT_TABLE table=test-table [id=e9aa3ab074c8485da12857a88029cf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:15.758587 4718 tablet_service.cc:1467] Processing CreateTablet for tablet d82946580e6f4408810d2d1af6bc69d5 (DEFAULT_TABLE table=test-table [id=e9aa3ab074c8485da12857a88029cf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:15.758739 4579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82946580e6f4408810d2d1af6bc69d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:15.759125 4718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82946580e6f4408810d2d1af6bc69d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:15.762485 4857 tablet_service.cc:1467] Processing CreateTablet for tablet d82946580e6f4408810d2d1af6bc69d5 (DEFAULT_TABLE table=test-table [id=e9aa3ab074c8485da12857a88029cf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:15.762997 4857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82946580e6f4408810d2d1af6bc69d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:15.765336 4950 tablet_bootstrap.cc:492] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Bootstrap starting.
I20260926 19:39:15.766837 4950 tablet_bootstrap.cc:654] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:15.783982 4948 tablet_bootstrap.cc:492] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d: Bootstrap starting.
I20260926 19:39:15.785809 4948 tablet_bootstrap.cc:654] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:15.789788 4948 tablet_bootstrap.cc:492] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d: No bootstrap required, opened a new log
I20260926 19:39:15.789790 4947 tablet_bootstrap.cc:492] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842: Bootstrap starting.
I20260926 19:39:15.789834 4950 tablet_bootstrap.cc:492] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: No bootstrap required, opened a new log
I20260926 19:39:15.789937 4948 ts_tablet_manager.cc:1397] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 19:39:15.789983 4950 ts_tablet_manager.cc:1397] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.010s
I20260926 19:39:15.790627 4950 raft_consensus.cc:348] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.790853 4950 raft_consensus.cc:374] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:15.790916 4950 raft_consensus.cc:729] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391ae9b2cdf347fcaecc27c6b7de7b5a, State: Initialized, Role: FOLLOWER
I20260926 19:39:15.791149 4950 consensus_queue.cc:260] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.791949 4947 tablet_bootstrap.cc:654] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:15.793888 4950 ts_tablet_manager.cc:1428] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260926 19:39:15.795589 4948 raft_consensus.cc:348] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.795821 4948 raft_consensus.cc:374] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:15.795897 4948 raft_consensus.cc:729] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba204f5b95274c3ea410a5f2c4da0c1d, State: Initialized, Role: FOLLOWER
I20260926 19:39:15.796629 4948 consensus_queue.cc:260] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.759136 5000 tablet_replica.cc:330] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: stopping tablet replica
I20260926 19:39:15.799237 5000 raft_consensus.cc:2229] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:15.797821 4948 ts_tablet_manager.cc:1428] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260926 19:39:15.800607 5000 raft_consensus.cc:2258] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:15.805904 4947 tablet_bootstrap.cc:492] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842: No bootstrap required, opened a new log
I20260926 19:39:15.806084 4947 ts_tablet_manager.cc:1397] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260926 19:39:15.806902 4947 raft_consensus.cc:348] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.807147 4947 raft_consensus.cc:374] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:15.807235 4947 raft_consensus.cc:729] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa94f8ca8f2478a9215341b8e12a842, State: Initialized, Role: FOLLOWER
I20260926 19:39:15.807523 4947 consensus_queue.cc:260] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.810326 4947 ts_tablet_manager.cc:1428] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 19:39:15.815510 5001 tablet_replica.cc:330] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: stopping tablet replica
I20260926 19:39:15.815824 5001 raft_consensus.cc:2229] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:39:15.816287 5001 raft_consensus.cc:2258] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:15.816526 5000 ts_tablet_manager.cc:1905] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:15.818032 5001 ts_tablet_manager.cc:1905] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:15.820506 4999 tablet_replica.cc:330] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a: stopping tablet replica
I20260926 19:39:15.820897 4999 raft_consensus.cc:2229] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:15.821228 4999 raft_consensus.cc:2258] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:15.823071 4999 ts_tablet_manager.cc:1905] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:15.825418 5001 ts_tablet_manager.cc:1918] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:39:15.825624 5001 log.cc:1196] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/e40a4f25194742d99ecef5220e2e6dda
I20260926 19:39:15.826277 5001 ts_tablet_manager.cc:1939] T e40a4f25194742d99ecef5220e2e6dda P 7aa94f8ca8f2478a9215341b8e12a842: Deleting consensus metadata
I20260926 19:39:15.827904 4430 catalog_manager.cc:4841] TS 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779): tablet e40a4f25194742d99ecef5220e2e6dda (table test-table [id=dd83266f7b284d5bbed1082e42daa524]) successfully deleted
I20260926 19:39:15.829247 5000 ts_tablet_manager.cc:1918] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:39:15.829404 5000 log.cc:1196] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/e40a4f25194742d99ecef5220e2e6dda
I20260926 19:39:15.829839 5000 ts_tablet_manager.cc:1939] T e40a4f25194742d99ecef5220e2e6dda P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting consensus metadata
I20260926 19:39:15.831355 4430 catalog_manager.cc:4841] TS ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66:32961): tablet e40a4f25194742d99ecef5220e2e6dda (table test-table [id=dd83266f7b284d5bbed1082e42daa524]) successfully deleted
I20260926 19:39:15.833882 4999 ts_tablet_manager.cc:1918] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:39:15.834014 4999 log.cc:1196] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/e40a4f25194742d99ecef5220e2e6dda
I20260926 19:39:15.834410 4999 ts_tablet_manager.cc:1939] T e40a4f25194742d99ecef5220e2e6dda P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting consensus metadata
I20260926 19:39:15.835640 4430 catalog_manager.cc:4841] TS 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883): tablet e40a4f25194742d99ecef5220e2e6dda (table test-table [id=dd83266f7b284d5bbed1082e42daa524]) successfully deleted
I20260926 19:39:15.881825 4953 raft_consensus.cc:482] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:15.882023 4953 raft_consensus.cc:504] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.883438 4953 leader_election.cc:288] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883), 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779)
I20260926 19:39:15.893558 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.883917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:41173 (local address 127.2.164.65:40883)
0926 19:39:15.884145 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.884151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.884748 (+ 597us) server_negotiation.cc:407] Connection header received
0926 19:39:15.885019 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.885025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.885103 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.885191 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:15.886020 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.887265 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.890760 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.892235 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.892376 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.892453 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.893049 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.893222 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:39:15.894546 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.883771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:40883 (local address 127.2.164.66:41173)
0926 19:39:15.884645 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.884663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.884851 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.885445 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.885451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.885479 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:15.885840 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.885852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.888204 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.888209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.890550 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.890566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.892654 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.892660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.892862 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.892896 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.894029 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.894178 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":605,"threads_started":1}
I20260926 19:39:15.897349 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82946580e6f4408810d2d1af6bc69d5" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" is_pre_election: true
I20260926 19:39:15.897616 4600 raft_consensus.cc:2454] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 0.
I20260926 19:39:15.898164 4672 leader_election.cc:302] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391ae9b2cdf347fcaecc27c6b7de7b5a, ba204f5b95274c3ea410a5f2c4da0c1d; no voters:
I20260926 19:39:15.898536 4953 raft_consensus.cc:2781] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:15.898640 4953 raft_consensus.cc:482] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:15.898720 4953 raft_consensus.cc:3037] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:15.901641 4953 raft_consensus.cc:504] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.902122 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.885129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:57379 (local address 127.2.164.67:43779)
0926 19:39:15.885341 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.885346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.895020 (+ 9674us) server_negotiation.cc:407] Connection header received
0926 19:39:15.895255 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.895261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.895334 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.895402 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:15.895934 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.897029 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.899747 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.901039 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.901154 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.901251 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.901741 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.901865 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 19:39:15.902287 4953 leader_election.cc:288] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 election: Requested vote from peers 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883), 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779)
I20260926 19:39:15.902830 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.884781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43779 (local address 127.2.164.66:57379)
0926 19:39:15.894927 (+ 10146us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.894946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.895112 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.895580 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.895585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.895610 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:15.895790 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.895799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.897233 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.897237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.899572 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.899585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.901371 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.901376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.901540 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.901571 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.902451 (+ 880us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.902572 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10048,"thread_start_us":11835,"threads_started":1}
I20260926 19:39:15.902863 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82946580e6f4408810d2d1af6bc69d5" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a"
I20260926 19:39:15.903054 4600 raft_consensus.cc:3037] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:15.903667 4877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82946580e6f4408810d2d1af6bc69d5" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa94f8ca8f2478a9215341b8e12a842" is_pre_election: true
I20260926 19:39:15.903927 4877 raft_consensus.cc:2454] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 0.
I20260926 19:39:15.904573 4877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82946580e6f4408810d2d1af6bc69d5" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa94f8ca8f2478a9215341b8e12a842"
I20260926 19:39:15.904754 4877 raft_consensus.cc:3037] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:15.905961 4600 raft_consensus.cc:2454] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 1.
I20260926 19:39:15.906448 4672 leader_election.cc:302] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391ae9b2cdf347fcaecc27c6b7de7b5a, ba204f5b95274c3ea410a5f2c4da0c1d; no voters:
I20260926 19:39:15.906745 4953 raft_consensus.cc:2781] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:15.907832 4877 raft_consensus.cc:2454] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 1.
I20260926 19:39:15.908861 4953 raft_consensus.cc:686] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 LEADER]: Becoming Leader. State: Replica: ba204f5b95274c3ea410a5f2c4da0c1d, State: Running, Role: LEADER
I20260926 19:39:15.909312 4953 consensus_queue.cc:237] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:15.912551 4442 catalog_manager.cc:5495] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d reported cstate change: term changed from 0 to 1, leader changed from <none> to ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66). New cstate: current_term: 1 leader_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } health_report { overall_health: UNKNOWN } } }
I20260926 19:39:15.935614 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.930240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.2.164.66:32961)
0926 19:39:15.930390 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.930395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.930410 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:39:15.930576 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.930580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.930674 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.930806 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:15.931134 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.932159 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.933565 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.934025 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.934134 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.934526 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.935058 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.935198 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.935267 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.935346 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:39:15.950212 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.944217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:47581 (local address 127.2.164.65:40883)
0926 19:39:15.944405 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.944411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.945727 (+ 1316us) server_negotiation.cc:407] Connection header received
0926 19:39:15.945901 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.945910 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.946007 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.946091 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:15.946523 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.947300 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.948565 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.949015 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.949120 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.949541 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.949644 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.949728 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.949846 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.949924 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:39:15.950284 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.944209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:40883 (local address 127.2.164.66:47581)
0926 19:39:15.945632 (+ 1423us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.945647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.945775 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.946101 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.946106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.946117 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:39:15.946390 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.946398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.947418 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.947424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.948438 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.948449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.949209 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.949215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.949336 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.949385 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.949741 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.949752 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.949818 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.949942 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"thread_start_us":1110,"threads_started":1}
I20260926 19:39:15.960764 4600 raft_consensus.cc:1261] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Refusing update from remote peer ba204f5b95274c3ea410a5f2c4da0c1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:15.960764 4877 raft_consensus.cc:1261] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Refusing update from remote peer ba204f5b95274c3ea410a5f2c4da0c1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:15.961412 4953 consensus_queue.cc:1035] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:15.961843 5007 consensus_queue.cc:1035] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:15.992739 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.986637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32961 (local address 127.2.164.65:55407)
0926 19:39:15.986882 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:15.986904 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:15.987045 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:15.987519 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:15.987524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:15.987536 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:39:15.987715 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.987728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.989046 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.989051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.990579 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:15.990593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.991553 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.991559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:15.991735 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.991782 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.992224 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.992235 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 19:39:15.992305 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 19:39:15.992443 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 19:39:15.993296 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:15.986693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:55407 (local address 127.2.164.66:32961)
0926 19:39:15.986852 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:15.986858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:15.986987 (+ 129us) server_negotiation.cc:407] Connection header received
0926 19:39:15.987181 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:15.987186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:15.987287 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:15.987370 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:15.987885 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.988897 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.990735 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:15.991334 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:15.991444 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:15.991960 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:15.992089 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:15.992175 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:15.992938 (+ 763us) server_negotiation.cc:299] Negotiation successful
0926 19:39:15.993014 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 19:39:15.994554 4976 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 19:39:17.608749 4443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:17.608963 4443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:17.612798 4443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4: Sending DeleteTablet for 3 replicas of tablet d82946580e6f4408810d2d1af6bc69d5
I20260926 19:39:17.613507 4580 tablet_service.cc:1514] Processing DeleteTablet for tablet d82946580e6f4408810d2d1af6bc69d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:17 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:42974
I20260926 19:39:17.613682 4858 tablet_service.cc:1514] Processing DeleteTablet for tablet d82946580e6f4408810d2d1af6bc69d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:17 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:50332
I20260926 19:39:17.613902 4718 tablet_service.cc:1514] Processing DeleteTablet for tablet d82946580e6f4408810d2d1af6bc69d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:17 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:40416
I20260926 19:39:17.622133 5037 tablet_replica.cc:330] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: stopping tablet replica
I20260926 19:39:17.622363 5037 raft_consensus.cc:2229] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:17.622615 5037 raft_consensus.cc:2258] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:17.623730 5037 ts_tablet_manager.cc:1905] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:17.624507 5039 tablet_replica.cc:330] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d: stopping tablet replica
I20260926 19:39:17.624693 5039 raft_consensus.cc:2229] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:39:17.624748 5038 tablet_replica.cc:330] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842: stopping tablet replica
I20260926 19:39:17.624912 5038 raft_consensus.cc:2229] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:17.625128 5038 raft_consensus.cc:2258] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:17.625294 5039 raft_consensus.cc:2258] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:17.626065 5038 ts_tablet_manager.cc:1905] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:17.626260 5039 ts_tablet_manager.cc:1905] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:17.631239 5037 ts_tablet_manager.cc:1918] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:39:17.631440 5037 log.cc:1196] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/d82946580e6f4408810d2d1af6bc69d5
I20260926 19:39:17.631984 5037 ts_tablet_manager.cc:1939] T d82946580e6f4408810d2d1af6bc69d5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting consensus metadata
I20260926 19:39:17.632210 5038 ts_tablet_manager.cc:1918] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:39:17.632339 5038 log.cc:1196] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/d82946580e6f4408810d2d1af6bc69d5
I20260926 19:39:17.632726 5038 ts_tablet_manager.cc:1939] T d82946580e6f4408810d2d1af6bc69d5 P 7aa94f8ca8f2478a9215341b8e12a842: Deleting consensus metadata
I20260926 19:39:17.632766 5039 ts_tablet_manager.cc:1918] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:39:17.632885 5039 log.cc:1196] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/d82946580e6f4408810d2d1af6bc69d5
I20260926 19:39:17.633348 5039 ts_tablet_manager.cc:1939] T d82946580e6f4408810d2d1af6bc69d5 P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting consensus metadata
I20260926 19:39:17.633487 4430 catalog_manager.cc:4841] TS 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883): tablet d82946580e6f4408810d2d1af6bc69d5 (table test-table [id=e9aa3ab074c8485da12857a88029cf68]) successfully deleted
I20260926 19:39:17.634271 4430 catalog_manager.cc:4841] TS ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66:32961): tablet d82946580e6f4408810d2d1af6bc69d5 (table test-table [id=e9aa3ab074c8485da12857a88029cf68]) successfully deleted
I20260926 19:39:17.636529 4430 catalog_manager.cc:4841] TS 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779): tablet d82946580e6f4408810d2d1af6bc69d5 (table test-table [id=e9aa3ab074c8485da12857a88029cf68]) successfully deleted
Sep 26 19:39:17 dist-test-slave-mrw4 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 19:39:17 dist-test-slave-mrw4 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:17.667958 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:17.650994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.2.164.126:42851)
0926 19:39:17.651533 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:17.651537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:17.651560 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:39:17.651723 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:17.651726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:17.651783 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:17.651877 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:17.653080 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:17.653907 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:17.655203 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:17.655583 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:17.655674 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:17.663558 (+ 7884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:17.663563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:17.663567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:17.663598 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:17.665918 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:17.666455 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:17.666458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:17.666461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:17.666570 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:17.666858 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:17.666861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:17.666863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:17.667098 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:17.667255 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:17.667487 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 19:39:17.667636 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":290,"threads_started":1}
I20260926 19:39:17.671145 4443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:39:17.671706 4443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:17.677642 4580 tablet_service.cc:1467] Processing CreateTablet for tablet 533ea1cda247480fa2c8827bdd5e2ba5 (DEFAULT_TABLE table=test-table [id=f670a0f81cb64fe9969f2073eadcd12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:17.677681 4718 tablet_service.cc:1467] Processing CreateTablet for tablet 533ea1cda247480fa2c8827bdd5e2ba5 (DEFAULT_TABLE table=test-table [id=f670a0f81cb64fe9969f2073eadcd12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:17.678064 4580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533ea1cda247480fa2c8827bdd5e2ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:17.678081 4718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533ea1cda247480fa2c8827bdd5e2ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:17.678120 4858 tablet_service.cc:1467] Processing CreateTablet for tablet 533ea1cda247480fa2c8827bdd5e2ba5 (DEFAULT_TABLE table=test-table [id=f670a0f81cb64fe9969f2073eadcd12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:17.678427 4858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533ea1cda247480fa2c8827bdd5e2ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:17.682821 5048 tablet_bootstrap.cc:492] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842: Bootstrap starting.
I20260926 19:39:17.684237 5048 tablet_bootstrap.cc:654] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:17.684769 5050 tablet_bootstrap.cc:492] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Bootstrap starting.
I20260926 19:39:17.685021 5049 tablet_bootstrap.cc:492] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d: Bootstrap starting.
I20260926 19:39:17.686407 5050 tablet_bootstrap.cc:654] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:17.686650 5049 tablet_bootstrap.cc:654] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:17.687077 5048 tablet_bootstrap.cc:492] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842: No bootstrap required, opened a new log
I20260926 19:39:17.687197 5048 ts_tablet_manager.cc:1397] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 19:39:17.687923 5048 raft_consensus.cc:348] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:17.688170 5048 raft_consensus.cc:374] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:17.688262 5048 raft_consensus.cc:729] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa94f8ca8f2478a9215341b8e12a842, State: Initialized, Role: FOLLOWER
I20260926 19:39:17.688421 5049 tablet_bootstrap.cc:492] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d: No bootstrap required, opened a new log
I20260926 19:39:17.688535 5049 ts_tablet_manager.cc:1397] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 19:39:17.688527 5048 consensus_queue.cc:260] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:17.689267 5049 raft_consensus.cc:348] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:17.689440 5049 raft_consensus.cc:374] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:17.689527 5049 raft_consensus.cc:729] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba204f5b95274c3ea410a5f2c4da0c1d, State: Initialized, Role: FOLLOWER
I20260926 19:39:17.689738 5049 consensus_queue.cc:260] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:17.690119 5049 ts_tablet_manager.cc:1428] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:17.690852 5048 ts_tablet_manager.cc:1428] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 19:39:17.691797 5050 tablet_bootstrap.cc:492] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: No bootstrap required, opened a new log
I20260926 19:39:17.691936 5050 ts_tablet_manager.cc:1397] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260926 19:39:17.692621 5050 raft_consensus.cc:348] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:17.692838 5050 raft_consensus.cc:374] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:17.692936 5050 raft_consensus.cc:729] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391ae9b2cdf347fcaecc27c6b7de7b5a, State: Initialized, Role: FOLLOWER
I20260926 19:39:17.693185 5050 consensus_queue.cc:260] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:17.693832 5050 ts_tablet_manager.cc:1428] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 19:39:17.904488 5031 raft_consensus.cc:482] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:17.904683 5031 raft_consensus.cc:504] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:17.905385 5031 leader_election.cc:288] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883), 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779)
I20260926 19:39:17.905946 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533ea1cda247480fa2c8827bdd5e2ba5" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" is_pre_election: true
I20260926 19:39:17.906004 4878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533ea1cda247480fa2c8827bdd5e2ba5" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa94f8ca8f2478a9215341b8e12a842" is_pre_election: true
I20260926 19:39:17.906158 4600 raft_consensus.cc:2454] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 0.
I20260926 19:39:17.906245 4878 raft_consensus.cc:2454] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 0.
I20260926 19:39:17.906589 4672 leader_election.cc:302] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391ae9b2cdf347fcaecc27c6b7de7b5a, ba204f5b95274c3ea410a5f2c4da0c1d; no voters:
I20260926 19:39:17.906827 5031 raft_consensus.cc:2781] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:17.906926 5031 raft_consensus.cc:482] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:17.907003 5031 raft_consensus.cc:3037] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:17.908898 5031 raft_consensus.cc:504] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:17.909382 5031 leader_election.cc:288] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 election: Requested vote from peers 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883), 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779)
I20260926 19:39:17.909768 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533ea1cda247480fa2c8827bdd5e2ba5" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a"
I20260926 19:39:17.909899 4878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533ea1cda247480fa2c8827bdd5e2ba5" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa94f8ca8f2478a9215341b8e12a842"
I20260926 19:39:17.909996 4600 raft_consensus.cc:3037] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:17.910068 4878 raft_consensus.cc:3037] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:17.912755 4600 raft_consensus.cc:2454] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 1.
I20260926 19:39:17.912767 4878 raft_consensus.cc:2454] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 1.
I20260926 19:39:17.913120 4672 leader_election.cc:302] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aa94f8ca8f2478a9215341b8e12a842, ba204f5b95274c3ea410a5f2c4da0c1d; no voters:
I20260926 19:39:17.913398 5031 raft_consensus.cc:2781] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:17.913520 5031 raft_consensus.cc:686] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 LEADER]: Becoming Leader. State: Replica: ba204f5b95274c3ea410a5f2c4da0c1d, State: Running, Role: LEADER
I20260926 19:39:17.913781 5031 consensus_queue.cc:237] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:17.915989 4442 catalog_manager.cc:5495] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d reported cstate change: term changed from 0 to 1, leader changed from <none> to ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66). New cstate: current_term: 1 leader_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } health_report { overall_health: UNKNOWN } } }
I20260926 19:39:17.991122 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:17.986312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.2.164.66:32961)
0926 19:39:17.986897 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:17.986902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:17.986919 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:39:17.986965 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:17.986969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:17.987024 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:17.987105 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:17.987485 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:17.988222 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:17.989510 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:17.989932 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:17.990037 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:17.990431 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:17.990525 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:17.990602 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:17.990757 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 19:39:17.990830 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":287,"threads_started":1}
I20260926 19:39:17.999225 4878 raft_consensus.cc:1261] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Refusing update from remote peer ba204f5b95274c3ea410a5f2c4da0c1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:17.999297 4600 raft_consensus.cc:1261] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Refusing update from remote peer ba204f5b95274c3ea410a5f2c4da0c1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:17.999718 5031 consensus_queue.cc:1035] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:17.999984 5014 consensus_queue.cc:1035] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:18.004756 5016 mvcc.cc:205] Tried to move back new op lower bound from 7333689581560844288 to 7333689581216432128. Current Snapshot: MvccSnapshot[applied={T|T < 7333689581560844288}]
I20260926 19:39:18.037407 4442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:18.037621 4442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:18.040200 4442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4: Sending DeleteTablet for 3 replicas of tablet 533ea1cda247480fa2c8827bdd5e2ba5
I20260926 19:39:18.040832 4580 tablet_service.cc:1514] Processing DeleteTablet for tablet 533ea1cda247480fa2c8827bdd5e2ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:18 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:42974
I20260926 19:39:18.041029 4858 tablet_service.cc:1514] Processing DeleteTablet for tablet 533ea1cda247480fa2c8827bdd5e2ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:18 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:50332
I20260926 19:39:18.041086 5037 tablet_replica.cc:330] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: stopping tablet replica
I20260926 19:39:18.041290 5038 tablet_replica.cc:330] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842: stopping tablet replica
I20260926 19:39:18.041291 5037 raft_consensus.cc:2229] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:18.041467 5037 raft_consensus.cc:2258] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:18.041486 5038 raft_consensus.cc:2229] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:18.041658 5038 raft_consensus.cc:2258] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:18.041684 4718 tablet_service.cc:1514] Processing DeleteTablet for tablet 533ea1cda247480fa2c8827bdd5e2ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:18 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:40416
I20260926 19:39:18.042084 5039 tablet_replica.cc:330] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d: stopping tablet replica
I20260926 19:39:18.042264 5039 raft_consensus.cc:2229] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:39:18.042194 4442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 19:39:18.042548 5039 raft_consensus.cc:2258] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:18.042667 5038 ts_tablet_manager.cc:1905] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:18.043426 5037 ts_tablet_manager.cc:1905] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:18.043470 5039 ts_tablet_manager.cc:1905] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 19:39:18.048645 4442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:18.050516 5037 ts_tablet_manager.cc:1918] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:39:18.050686 5037 log.cc:1196] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/533ea1cda247480fa2c8827bdd5e2ba5
I20260926 19:39:18.050856 5039 ts_tablet_manager.cc:1918] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:39:18.050869 5038 ts_tablet_manager.cc:1918] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:39:18.051002 5039 log.cc:1196] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/533ea1cda247480fa2c8827bdd5e2ba5
I20260926 19:39:18.051007 5038 log.cc:1196] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/533ea1cda247480fa2c8827bdd5e2ba5
I20260926 19:39:18.051239 5037 ts_tablet_manager.cc:1939] T 533ea1cda247480fa2c8827bdd5e2ba5 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting consensus metadata
I20260926 19:39:18.051357 5039 ts_tablet_manager.cc:1939] T 533ea1cda247480fa2c8827bdd5e2ba5 P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting consensus metadata
I20260926 19:39:18.051393 5038 ts_tablet_manager.cc:1939] T 533ea1cda247480fa2c8827bdd5e2ba5 P 7aa94f8ca8f2478a9215341b8e12a842: Deleting consensus metadata
I20260926 19:39:18.052234 4430 catalog_manager.cc:4841] TS 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779): tablet 533ea1cda247480fa2c8827bdd5e2ba5 (table test-table [id=f670a0f81cb64fe9969f2073eadcd12e]) successfully deleted
I20260926 19:39:18.052454 4430 catalog_manager.cc:4841] TS 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883): tablet 533ea1cda247480fa2c8827bdd5e2ba5 (table test-table [id=f670a0f81cb64fe9969f2073eadcd12e]) successfully deleted
I20260926 19:39:18.052627 4430 catalog_manager.cc:4841] TS ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66:32961): tablet 533ea1cda247480fa2c8827bdd5e2ba5 (table test-table [id=f670a0f81cb64fe9969f2073eadcd12e]) successfully deleted
I20260926 19:39:18.059298 4580 tablet_service.cc:1467] Processing CreateTablet for tablet 07eefaae52834fe68ca24036b106d250 (DEFAULT_TABLE table=test-table [id=2d47375eadcd4eedb6c7292d32741e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:18.059338 4858 tablet_service.cc:1467] Processing CreateTablet for tablet 07eefaae52834fe68ca24036b106d250 (DEFAULT_TABLE table=test-table [id=2d47375eadcd4eedb6c7292d32741e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:18.059669 4580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07eefaae52834fe68ca24036b106d250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:18.059690 4858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07eefaae52834fe68ca24036b106d250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:18.064266 5050 tablet_bootstrap.cc:492] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Bootstrap starting.
I20260926 19:39:18.065521 4718 tablet_service.cc:1467] Processing CreateTablet for tablet 07eefaae52834fe68ca24036b106d250 (DEFAULT_TABLE table=test-table [id=2d47375eadcd4eedb6c7292d32741e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:18.065871 5050 tablet_bootstrap.cc:654] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:18.065884 4718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07eefaae52834fe68ca24036b106d250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:18.072873 5048 tablet_bootstrap.cc:492] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842: Bootstrap starting.
I20260926 19:39:18.073726 5049 tablet_bootstrap.cc:492] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d: Bootstrap starting.
I20260926 19:39:18.075160 5049 tablet_bootstrap.cc:654] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:18.077572 5049 tablet_bootstrap.cc:492] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d: No bootstrap required, opened a new log
I20260926 19:39:18.077706 5049 ts_tablet_manager.cc:1397] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 19:39:18.078338 5049 raft_consensus.cc:348] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:18.078555 5049 raft_consensus.cc:374] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:18.078624 5049 raft_consensus.cc:729] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba204f5b95274c3ea410a5f2c4da0c1d, State: Initialized, Role: FOLLOWER
I20260926 19:39:18.078822 5049 consensus_queue.cc:260] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:18.079285 5049 ts_tablet_manager.cc:1428] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 19:39:18.079424 5048 tablet_bootstrap.cc:654] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:18.086486 5048 tablet_bootstrap.cc:492] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842: No bootstrap required, opened a new log
I20260926 19:39:18.086758 5048 ts_tablet_manager.cc:1397] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20260926 19:39:18.087600 5048 raft_consensus.cc:348] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:18.087962 5048 raft_consensus.cc:374] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:18.088081 5048 raft_consensus.cc:729] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa94f8ca8f2478a9215341b8e12a842, State: Initialized, Role: FOLLOWER
I20260926 19:39:18.088382 5048 consensus_queue.cc:260] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:18.088979 5048 ts_tablet_manager.cc:1428] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 19:39:18.091564 5050 tablet_bootstrap.cc:492] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a: No bootstrap required, opened a new log
I20260926 19:39:18.091717 5050 ts_tablet_manager.cc:1397] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.016s
I20260926 19:39:18.092474 5050 raft_consensus.cc:348] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:18.092666 5050 raft_consensus.cc:374] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:18.092744 5050 raft_consensus.cc:729] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391ae9b2cdf347fcaecc27c6b7de7b5a, State: Initialized, Role: FOLLOWER
I20260926 19:39:18.092972 5050 consensus_queue.cc:260] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:18.093532 5050 ts_tablet_manager.cc:1428] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 19:39:18.271656 5014 raft_consensus.cc:482] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:18.271916 5014 raft_consensus.cc:504] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:18.272675 5014 leader_election.cc:288] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883), 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779)
I20260926 19:39:18.273216 4599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07eefaae52834fe68ca24036b106d250" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" is_pre_election: true
I20260926 19:39:18.273386 4878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07eefaae52834fe68ca24036b106d250" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa94f8ca8f2478a9215341b8e12a842" is_pre_election: true
I20260926 19:39:18.273492 4599 raft_consensus.cc:2454] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 0.
I20260926 19:39:18.273656 4878 raft_consensus.cc:2454] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 0.
I20260926 19:39:18.273995 4672 leader_election.cc:302] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391ae9b2cdf347fcaecc27c6b7de7b5a, ba204f5b95274c3ea410a5f2c4da0c1d; no voters:
I20260926 19:39:18.274300 5014 raft_consensus.cc:2781] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:18.274397 5014 raft_consensus.cc:482] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:18.274498 5014 raft_consensus.cc:3037] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:18.276441 5014 raft_consensus.cc:504] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:18.276961 5014 leader_election.cc:288] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 election: Requested vote from peers 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883), 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779)
I20260926 19:39:18.277467 4878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07eefaae52834fe68ca24036b106d250" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa94f8ca8f2478a9215341b8e12a842"
I20260926 19:39:18.277447 4599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07eefaae52834fe68ca24036b106d250" candidate_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a"
I20260926 19:39:18.277633 4878 raft_consensus.cc:3037] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:18.277635 4599 raft_consensus.cc:3037] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:18.279600 4878 raft_consensus.cc:2454] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 1.
I20260926 19:39:18.279917 4672 leader_election.cc:302] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aa94f8ca8f2478a9215341b8e12a842, ba204f5b95274c3ea410a5f2c4da0c1d; no voters:
I20260926 19:39:18.280139 5014 raft_consensus.cc:2781] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:18.280305 5014 raft_consensus.cc:686] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 LEADER]: Becoming Leader. State: Replica: ba204f5b95274c3ea410a5f2c4da0c1d, State: Running, Role: LEADER
I20260926 19:39:18.280582 4599 raft_consensus.cc:2454] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba204f5b95274c3ea410a5f2c4da0c1d in term 1.
I20260926 19:39:18.280766 5014 consensus_queue.cc:237] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } }
I20260926 19:39:18.283384 4443 catalog_manager.cc:5495] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d reported cstate change: term changed from 0 to 1, leader changed from <none> to ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66). New cstate: current_term: 1 leader_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba204f5b95274c3ea410a5f2c4da0c1d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 } health_report { overall_health: UNKNOWN } } }
I20260926 19:39:18.316906 4878 raft_consensus.cc:1261] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Refusing update from remote peer ba204f5b95274c3ea410a5f2c4da0c1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:18.316977 4599 raft_consensus.cc:1261] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Refusing update from remote peer ba204f5b95274c3ea410a5f2c4da0c1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:18.317406 5014 consensus_queue.cc:1035] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aa94f8ca8f2478a9215341b8e12a842" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:18.317641 5031 consensus_queue.cc:1035] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "391ae9b2cdf347fcaecc27c6b7de7b5a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 40883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 19:39:18.334103 5021 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 19:39:19.903542 4442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:19.903744 4442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:19.907178 4442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b5f64af1c364873b5814dccc66161a4: Sending DeleteTablet for 3 replicas of tablet 07eefaae52834fe68ca24036b106d250
I20260926 19:39:19.907754 4580 tablet_service.cc:1514] Processing DeleteTablet for tablet 07eefaae52834fe68ca24036b106d250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:19 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:42974
I20260926 19:39:19.907852 4718 tablet_service.cc:1514] Processing DeleteTablet for tablet 07eefaae52834fe68ca24036b106d250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:19 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:40416
I20260926 19:39:19.907925 4858 tablet_service.cc:1514] Processing DeleteTablet for tablet 07eefaae52834fe68ca24036b106d250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:19 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:50332
I20260926 19:39:19.908504 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 4515
I20260926 19:39:19.922911 5091 tablet_replica.cc:330] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a: stopping tablet replica
I20260926 19:39:19.923280 5092 tablet_replica.cc:330] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d: stopping tablet replica
I20260926 19:39:19.923241 5091 raft_consensus.cc:2229] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:19.923488 5091 raft_consensus.cc:2258] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:19.923604 5092 raft_consensus.cc:2229] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:39:19.924010 5092 raft_consensus.cc:2258] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:19.924516 5091 ts_tablet_manager.cc:1905] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:19.925133 5092 ts_tablet_manager.cc:1905] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:19.931303 5091 ts_tablet_manager.cc:1918] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:39:19.931550 5091 log.cc:1196] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/07eefaae52834fe68ca24036b106d250
I20260926 19:39:19.932011 5093 tablet_replica.cc:330] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842: stopping tablet replica
I20260926 19:39:19.932065 5092 ts_tablet_manager.cc:1918] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:39:19.932200 5092 log.cc:1196] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/07eefaae52834fe68ca24036b106d250
I20260926 19:39:19.932206 5093 raft_consensus.cc:2229] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:19.932365 5091 ts_tablet_manager.cc:1939] T 07eefaae52834fe68ca24036b106d250 P 391ae9b2cdf347fcaecc27c6b7de7b5a: Deleting consensus metadata
I20260926 19:39:19.932447 5093 raft_consensus.cc:2258] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:19.932618 5092 ts_tablet_manager.cc:1939] T 07eefaae52834fe68ca24036b106d250 P ba204f5b95274c3ea410a5f2c4da0c1d: Deleting consensus metadata
I20260926 19:39:19.933410 4430 catalog_manager.cc:4841] TS 391ae9b2cdf347fcaecc27c6b7de7b5a (127.2.164.65:40883): tablet 07eefaae52834fe68ca24036b106d250 (table test-table [id=2d47375eadcd4eedb6c7292d32741e3f]) successfully deleted
I20260926 19:39:19.933712 4430 catalog_manager.cc:4841] TS ba204f5b95274c3ea410a5f2c4da0c1d (127.2.164.66:32961): tablet 07eefaae52834fe68ca24036b106d250 (table test-table [id=2d47375eadcd4eedb6c7292d32741e3f]) successfully deleted
I20260926 19:39:19.933713 5093 ts_tablet_manager.cc:1905] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 19:39:19 dist-test-slave-mrw4 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:19.938115 5093 ts_tablet_manager.cc:1918] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:39:19.938225 5093 log.cc:1196] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/07eefaae52834fe68ca24036b106d250
I20260926 19:39:19.938539 5093 ts_tablet_manager.cc:1939] T 07eefaae52834fe68ca24036b106d250 P 7aa94f8ca8f2478a9215341b8e12a842: Deleting consensus metadata
I20260926 19:39:19.939321 4430 catalog_manager.cc:4841] TS 7aa94f8ca8f2478a9215341b8e12a842 (127.2.164.67:43779): tablet 07eefaae52834fe68ca24036b106d250 (table test-table [id=2d47375eadcd4eedb6c7292d32741e3f]) successfully deleted
I20260926 19:39:19.941244 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:19.909511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.2.164.65:40883)
0926 19:39:19.925257 (+ 15746us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:19.925263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:19.925285 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:39:19.925336 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:19.925340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:19.925412 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:19.925517 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:19.927364 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:19.928379 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:19.930199 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:19.930839 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:19.930947 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:19.935956 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:19.935970 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:19.935987 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:19.936015 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:19.938862 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:19.939550 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:19.939557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:19.939560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:19.939699 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:19.940062 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:19.940065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:19.940067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:19.940369 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:19.940585 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:19.940757 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 19:39:19.940906 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15595,"thread_start_us":15525,"threads_started":1}
W20260926 19:39:19.941802 4645 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35646
W20260926 19:39:19.942328 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 4515: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:39:19.942497 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 4515
I20260926 19:39:19.960357 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 4654
Sep 26 19:39:19 dist-test-slave-mrw4 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:39:19.973636 5096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:19.960960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.2.164.66:32961)
0926 19:39:19.961445 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:19.961449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:19.961464 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:39:19.961515 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:19.961518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:19.961573 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:19.961658 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:19.962745 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:19.963454 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:19.964615 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:19.965019 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:19.965109 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:19.968893 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:19.968906 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:19.968923 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:19.968959 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:19.971468 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:19.971993 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:19.972000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:19.972003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:19.972110 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:19.972359 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:19.972362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:19.972364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:19.972681 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:19.972878 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:19.973059 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 19:39:19.973315 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":246,"threads_started":1}
W20260926 19:39:19.974273 4784 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39150
W20260926 19:39:19.974853 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 4654: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:39:19.974965 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 4654
I20260926 19:39:19.992322 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 4793
Sep 26 19:39:19 dist-test-slave-mrw4 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:39:20.006302 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:19.993094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.2.164.67:43779)
0926 19:39:19.993609 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:19.993613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:19.993629 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:39:19.993681 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:19.993685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:19.993741 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:19.993825 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:19.995123 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:19.995869 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:19.997025 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:19.997452 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:19.997536 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:20.001431 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:20.001447 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:20.001463 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:20.001499 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:20.004072 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:20.004652 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:20.004659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:20.004661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:20.004790 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:20.005067 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:20.005070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:20.005072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:20.005403 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:20.005607 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:20.005818 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 19:39:20.005969 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":272,"threads_started":1}
W20260926 19:39:20.006953 4923 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58244
W20260926 19:39:20.007478 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 4793: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:39:20.007643 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 4793
I20260926 19:39:20.025204 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 4412
I20260926 19:39:20.026399 4484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:20.138904 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 4412
2026-09-26T19:39:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8191 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:20 dist-test-slave-mrw4 krb5kdc[5101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:20 dist-test-slave-mrw4 krb5kdc[5101](info): set up 2 sockets
Sep 26 19:39:20 dist-test-slave-mrw4 krb5kdc[5101](info): commencing operation
krb5kdc: starting...
W20260926 19:39:21.129019 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:21 dist-test-slave-mrw4 krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:39:21.291954 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38933
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:38933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:21.587184 5114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:21.587471 5114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:21.587598 5114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:21.594684 5114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:21.594791 5114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:21.594866 5114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:21.594938 5114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:21.594997 5114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:21.604463 5114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:38933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:21.605733 5114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:21.607038 5114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:21.617942 5120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:21.618021 5119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:21.617995 5122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:21.619065 5114 server_base.cc:1034] running on GCE node
I20260926 19:39:21.619791 5114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:21.621057 5114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:21.622283 5114 hybrid_clock.cc:648] HybridClock initialized: now 1790451561622231 us; error 42 us; skew 500 ppm
I20260926 19:39:21.622602 5114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:21 dist-test-slave-mrw4 krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451561, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:21.625480 5114 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:39:21.626806 5114 webserver.cc:460] Webserver started at http://127.2.164.126:39715/ using document root <none> and password file <none>
I20260926 19:39:21.627419 5114 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:21.627513 5114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:21.627763 5114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:21.630505 5114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "e5ce99d2e37e4ec6930c27abf98d97e4"
format_stamp: "Formatted at 2026-09-26 19:39:21 on dist-test-slave-mrw4"
server_key: "462dee92c36e3793ce18702bf7ebe8ea"
server_key_iv: "2899888a610822b535b25defaf975964"
server_key_version: "encryptionkey@0"
I20260926 19:39:21.631204 5114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "e5ce99d2e37e4ec6930c27abf98d97e4"
format_stamp: "Formatted at 2026-09-26 19:39:21 on dist-test-slave-mrw4"
server_key: "462dee92c36e3793ce18702bf7ebe8ea"
server_key_iv: "2899888a610822b535b25defaf975964"
server_key_version: "encryptionkey@0"
I20260926 19:39:21.635833 5114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:39:21.638957 5129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:21.640139 5114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 19:39:21.640268 5114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "e5ce99d2e37e4ec6930c27abf98d97e4"
format_stamp: "Formatted at 2026-09-26 19:39:21 on dist-test-slave-mrw4"
server_key: "462dee92c36e3793ce18702bf7ebe8ea"
server_key_iv: "2899888a610822b535b25defaf975964"
server_key_version: "encryptionkey@0"
I20260926 19:39:21.640412 5114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:21.652257 5114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:21.655766 5114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:21.655968 5114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:21.671211 5114 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:38933
I20260926 19:39:21.671229 5180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:38933 every 8 connection(s)
I20260926 19:39:21.672359 5114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:21.675010 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 5114
I20260926 19:39:21.675196 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:21.675746 5181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:21.675767 2705 external_mini_cluster.cc:1428] Setting key 6c07c4b8e9441db9e4325a01ddc1c2c0
I20260926 19:39:21.688580 5181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4: Bootstrap starting.
Sep 26 19:39:21 dist-test-slave-mrw4 krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:21.691390 5181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:21.692332 5181 log.cc:824] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:21.695019 5181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4: No bootstrap required, opened a new log
I20260926 19:39:21.696291 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:21.677467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.2.164.126:38933)
0926 19:39:21.678167 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:21.678177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:21.678218 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:39:21.679865 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:21.679885 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:21.680244 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:21.680545 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:21.682206 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:21.683409 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:21.685054 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:21.685778 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:21.685902 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:21.690036 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:21.690043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:21.690055 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:21.690097 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:21.692800 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:21.693429 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:21.693435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:21.693441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:21.693587 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:21.693951 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:21.693954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:21.693956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:21.694421 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:21.694620 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:21.695014 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 19:39:21.695393 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":312,"threads_started":1}
I20260926 19:39:21.702085 5181 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ce99d2e37e4ec6930c27abf98d97e4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38933 } }
I20260926 19:39:21.702386 5181 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:21.702484 5181 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ce99d2e37e4ec6930c27abf98d97e4, State: Initialized, Role: FOLLOWER
I20260926 19:39:21.703066 5181 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ce99d2e37e4ec6930c27abf98d97e4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38933 } }
I20260926 19:39:21.703266 5181 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:21.703351 5181 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:21.703465 5181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:21.705351 5181 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ce99d2e37e4ec6930c27abf98d97e4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38933 } }
I20260926 19:39:21.705744 5181 leader_election.cc:302] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5ce99d2e37e4ec6930c27abf98d97e4; no voters:
I20260926 19:39:21.706214 5181 leader_election.cc:288] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:21.706321 5186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:21.706887 5186 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [term 1 LEADER]: Becoming Leader. State: Replica: e5ce99d2e37e4ec6930c27abf98d97e4, State: Running, Role: LEADER
I20260926 19:39:21.707368 5186 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ce99d2e37e4ec6930c27abf98d97e4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38933 } }
I20260926 19:39:21.707890 5181 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:21.709815 5187 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5ce99d2e37e4ec6930c27abf98d97e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ce99d2e37e4ec6930c27abf98d97e4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38933 } } }
I20260926 19:39:21.710146 5187 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:21.710577 5188 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5ce99d2e37e4ec6930c27abf98d97e4. Latest consensus state: current_term: 1 leader_uuid: "e5ce99d2e37e4ec6930c27abf98d97e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ce99d2e37e4ec6930c27abf98d97e4" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38933 } } }
I20260926 19:39:21.710956 5188 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:21.712229 5193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:21.716311 5193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:21.723587 5193 catalog_manager.cc:1347] Generated new cluster ID: 963302cd647d4658941ff975469336be
I20260926 19:39:21.723680 5193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:21.732939 5193 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:21.734624 5193 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:21.745839 5193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5ce99d2e37e4ec6930c27abf98d97e4: Generated new TSK 0
I20260926 19:39:21.746730 5193 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:39:21.792836 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38933
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:22.092402 5207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:22.092689 5207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:22.092824 5207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:22.099771 5207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:22.099870 5207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:22.100018 5207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:22.109486 5207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:22.110620 5207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:22.111792 5207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:22.123411 5212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:22.123482 5213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:22.123456 5215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:22.124285 5207 server_base.cc:1034] running on GCE node
I20260926 19:39:22.124770 5207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:22.125656 5207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:22.126842 5207 hybrid_clock.cc:648] HybridClock initialized: now 1790451562126780 us; error 57 us; skew 500 ppm
I20260926 19:39:22.127118 5207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:22 dist-test-slave-mrw4 krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451562, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:22.129963 5207 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:39:22.131232 5207 webserver.cc:460] Webserver started at http://127.2.164.65:46117/ using document root <none> and password file <none>
I20260926 19:39:22.131798 5207 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:22.131888 5207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:22.132122 5207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:22.134773 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "04d51f7832c14896873ba041a1f2e21c"
format_stamp: "Formatted at 2026-09-26 19:39:22 on dist-test-slave-mrw4"
server_key: "1c20a1ab8cd15cd1bfe81069cfb727e4"
server_key_iv: "c962e91a7529fcd0729954be6ad002ea"
server_key_version: "encryptionkey@0"
I20260926 19:39:22.135443 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "04d51f7832c14896873ba041a1f2e21c"
format_stamp: "Formatted at 2026-09-26 19:39:22 on dist-test-slave-mrw4"
server_key: "1c20a1ab8cd15cd1bfe81069cfb727e4"
server_key_iv: "c962e91a7529fcd0729954be6ad002ea"
server_key_version: "encryptionkey@0"
I20260926 19:39:22.139952 5207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 19:39:22.142984 5222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:22.144183 5207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 19:39:22.144301 5207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "04d51f7832c14896873ba041a1f2e21c"
format_stamp: "Formatted at 2026-09-26 19:39:22 on dist-test-slave-mrw4"
server_key: "1c20a1ab8cd15cd1bfe81069cfb727e4"
server_key_iv: "c962e91a7529fcd0729954be6ad002ea"
server_key_version: "encryptionkey@0"
I20260926 19:39:22.144443 5207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:22.155788 5207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:22.159461 5207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:22.159667 5207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:22.160442 5207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:22.161638 5207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:22.161705 5207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:22.161784 5207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:22.161828 5207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:22.190062 5207 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:34543
I20260926 19:39:22.190135 5334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:34543 every 8 connection(s)
I20260926 19:39:22.191252 5207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:22.198575 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 5207
I20260926 19:39:22.198774 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:22.199492 2705 external_mini_cluster.cc:1428] Setting key 360a8b81a6fb76fb95c23a43e59d0dce
Sep 26 19:39:22 dist-test-slave-mrw4 krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451562, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:22.210317 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:22.193757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:36709 (local address 127.2.164.126:38933)
0926 19:39:22.193914 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:22.193919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:22.194984 (+ 1065us) server_negotiation.cc:407] Connection header received
0926 19:39:22.196675 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:22.196679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:22.196742 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:22.196840 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:22.198249 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:22.199270 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:22.201101 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:22.201631 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:22.201755 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:22.204895 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:22.204900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:22.204905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:22.204940 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:22.207450 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:22.208152 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:22.208156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:22.208158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:22.208278 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:22.208659 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:22.208664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:22.208667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:22.208934 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:22.209079 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:22.209767 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 19:39:22.209994 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:39:22.211371 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:22.194144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38933 (local address 127.2.164.65:36709)
0926 19:39:22.194819 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:22.194855 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:22.196435 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:22.197025 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:22.197033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:22.197555 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:22.198096 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:22.198114 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:22.199418 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:22.199422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:22.200983 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:22.200996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:22.201843 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:22.201848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:22.202053 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:22.202663 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:22.202685 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:22.204682 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:22.207597 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:22.207607 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:22.207621 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:22.208039 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:22.208376 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:22.208380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:22.208384 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:22.208562 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:22.209048 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:22.209056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:22.209427 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 19:39:22.209674 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 19:39:22.210196 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":259,"threads_started":1}
I20260926 19:39:22.212888 5335 heartbeater.cc:347] Connected to a master server at 127.2.164.126:38933
I20260926 19:39:22.213251 5335 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:22.214135 5335 heartbeater.cc:511] Master 127.2.164.126:38933 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260926 19:39:22.216337 5146 ts_manager.cc:194] Registered new tserver with Master: 04d51f7832c14896873ba041a1f2e21c (127.2.164.65:34543)
I20260926 19:39:22.217790 5146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:36709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:39:22.237363 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38933
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:22.535501 5340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:22.535773 5340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:22.535902 5340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:22.543030 5340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:22.543146 5340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:22.543298 5340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:22.553048 5340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:22.554220 5340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:22.555408 5340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:22.567205 5345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:22.567178 5346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:22.567320 5348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:22.568045 5340 server_base.cc:1034] running on GCE node
I20260926 19:39:22.568569 5340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:22.569463 5340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:22.570652 5340 hybrid_clock.cc:648] HybridClock initialized: now 1790451562570602 us; error 41 us; skew 500 ppm
I20260926 19:39:22.570920 5340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:22 dist-test-slave-mrw4 krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451562, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:22.573809 5340 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:39:22.575150 5340 webserver.cc:460] Webserver started at http://127.2.164.66:42291/ using document root <none> and password file <none>
I20260926 19:39:22.575760 5340 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:22.575867 5340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:22.576136 5340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:22.578807 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "7978ae5b585e48e9bc3eb2e486fa7b0c"
format_stamp: "Formatted at 2026-09-26 19:39:22 on dist-test-slave-mrw4"
server_key: "1fc46609fc58675222644c1fb229d9b8"
server_key_iv: "877891ab2ca0efd084dd3ef170fe3d98"
server_key_version: "encryptionkey@0"
I20260926 19:39:22.579506 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "7978ae5b585e48e9bc3eb2e486fa7b0c"
format_stamp: "Formatted at 2026-09-26 19:39:22 on dist-test-slave-mrw4"
server_key: "1fc46609fc58675222644c1fb229d9b8"
server_key_iv: "877891ab2ca0efd084dd3ef170fe3d98"
server_key_version: "encryptionkey@0"
I20260926 19:39:22.584043 5340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:39:22.587170 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:22.588399 5340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:22.588547 5340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "7978ae5b585e48e9bc3eb2e486fa7b0c"
format_stamp: "Formatted at 2026-09-26 19:39:22 on dist-test-slave-mrw4"
server_key: "1fc46609fc58675222644c1fb229d9b8"
server_key_iv: "877891ab2ca0efd084dd3ef170fe3d98"
server_key_version: "encryptionkey@0"
I20260926 19:39:22.588711 5340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:22.609337 5340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:22.612954 5340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:22.613250 5340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:22.614107 5340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:22.615319 5340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:22.615386 5340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:22.615468 5340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:22.615507 5340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:22.643826 5340 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:43859
I20260926 19:39:22.643877 5467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:43859 every 8 connection(s)
I20260926 19:39:22.645026 5340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:22.651110 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 5340
I20260926 19:39:22.651259 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:22.651651 2705 external_mini_cluster.cc:1428] Setting key 35ee4c23d6724d78084e66359803f392
Sep 26 19:39:22 dist-test-slave-mrw4 krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451562, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:22.663337 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:22.647483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:35519 (local address 127.2.164.126:38933)
0926 19:39:22.647689 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:22.647695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:22.648849 (+ 1154us) server_negotiation.cc:407] Connection header received
0926 19:39:22.650592 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:22.650596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:22.650660 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:22.650747 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:22.652204 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:22.652963 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:22.654476 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:22.655119 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:22.655236 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:22.658435 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:22.658439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:22.658442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:22.658472 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:22.660636 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:22.661293 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:22.661297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:22.661299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:22.661424 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:22.661801 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:22.661804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:22.661806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:22.662054 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:22.662173 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:22.662782 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 19:39:22.662980 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:39:22.664280 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:22.648006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38933 (local address 127.2.164.66:35519)
0926 19:39:22.648712 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:22.648749 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:22.650365 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:22.650925 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:22.650932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:22.651484 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:22.652054 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:22.652072 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:22.653129 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:22.653134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:22.654340 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:22.654351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:22.655286 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:22.655290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:22.655475 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:22.656062 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:22.656083 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:22.658251 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:22.660757 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:22.660766 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:22.660781 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:22.661194 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:22.661526 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:22.661531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:22.661535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:22.661698 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:22.662165 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:22.662172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:22.662458 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 19:39:22.662677 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 19:39:22.663130 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":271,"threads_started":1}
I20260926 19:39:22.665720 5468 heartbeater.cc:347] Connected to a master server at 127.2.164.126:38933
I20260926 19:39:22.665967 5468 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260926 19:39:22.666581 5468 heartbeater.cc:511] Master 127.2.164.126:38933 requested a full tablet report, sending...
I20260926 19:39:22.667852 5146 ts_manager.cc:194] Registered new tserver with Master: 7978ae5b585e48e9bc3eb2e486fa7b0c (127.2.164.66:43859)
I20260926 19:39:22.668562 5146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:35519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:39:22.700326 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38933
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:22.996917 5473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:22.997203 5473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:22.997318 5473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:23.004277 5473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:23.004372 5473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:23.004488 5473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:23.013969 5473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:38933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:23.015100 5473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:23.016314 5473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:23.028136 5479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:23.028209 5481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:23.028265 5478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:23.029212 5473 server_base.cc:1034] running on GCE node
I20260926 19:39:23.029753 5473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:23.030596 5473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:23.031788 5473 hybrid_clock.cc:648] HybridClock initialized: now 1790451563031734 us; error 50 us; skew 500 ppm
I20260926 19:39:23.032070 5473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:23 dist-test-slave-mrw4 krb5kdc[5101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451563, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:23.038477 5473 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:39:23.039855 5473 webserver.cc:460] Webserver started at http://127.2.164.67:46373/ using document root <none> and password file <none>
I20260926 19:39:23.040514 5473 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:23.040616 5473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:23.040898 5473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:23.043812 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "f97ae041294846e7bbff85ea112b4eac"
format_stamp: "Formatted at 2026-09-26 19:39:23 on dist-test-slave-mrw4"
server_key: "f22c8cae0319ec7df6e2c10f48724f83"
server_key_iv: "93b4e083f2f3e3fdbfd622192144b3f5"
server_key_version: "encryptionkey@0"
I20260926 19:39:23.044544 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "f97ae041294846e7bbff85ea112b4eac"
format_stamp: "Formatted at 2026-09-26 19:39:23 on dist-test-slave-mrw4"
server_key: "f22c8cae0319ec7df6e2c10f48724f83"
server_key_iv: "93b4e083f2f3e3fdbfd622192144b3f5"
server_key_version: "encryptionkey@0"
I20260926 19:39:23.049489 5473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 19:39:23.052618 5488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:23.053864 5473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:39:23.054003 5473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "f97ae041294846e7bbff85ea112b4eac"
format_stamp: "Formatted at 2026-09-26 19:39:23 on dist-test-slave-mrw4"
server_key: "f22c8cae0319ec7df6e2c10f48724f83"
server_key_iv: "93b4e083f2f3e3fdbfd622192144b3f5"
server_key_version: "encryptionkey@0"
I20260926 19:39:23.054173 5473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:23.074690 5473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:23.078292 5473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:23.078528 5473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:23.079384 5473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:23.080570 5473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:23.080647 5473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:23.080758 5473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:23.080808 5473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:23.109018 5473 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:44763
I20260926 19:39:23.109081 5600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:44763 every 8 connection(s)
I20260926 19:39:23.110270 5473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:23.114387 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 5473
I20260926 19:39:23.114563 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:23.114952 2705 external_mini_cluster.cc:1428] Setting key d806a6842933c657dcc8eb25625865a9
Sep 26 19:39:23 dist-test-slave-mrw4 krb5kdc[5101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451563, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:23.128194 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:23.112562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:36747 (local address 127.2.164.126:38933)
0926 19:39:23.112726 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:23.112731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:23.113908 (+ 1177us) server_negotiation.cc:407] Connection header received
0926 19:39:23.115780 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:23.115785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:23.115864 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:23.115946 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:23.117516 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:23.118203 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:23.119645 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:23.120035 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:23.120136 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:23.123184 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:23.123189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:23.123192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:23.123222 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:23.125568 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:23.126202 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:23.126206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:23.126208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:23.126315 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:23.126685 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:23.126689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:23.126691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:23.126931 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:23.127068 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:23.127674 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 19:39:23.127883 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:39:23.129114 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:23.113057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38933 (local address 127.2.164.67:36747)
0926 19:39:23.113758 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:23.113804 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:23.115502 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:23.116203 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:23.116214 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:23.116783 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:23.117364 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:23.117383 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:23.118334 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:23.118338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:23.119516 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:23.119528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:23.120227 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:23.120233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:23.120439 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:23.120992 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:23.121012 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:23.122966 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:23.125692 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:23.125700 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:23.125716 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:23.126078 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:23.126408 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:23.126413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:23.126417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:23.126578 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:23.127057 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:23.127065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:23.127351 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 19:39:23.127565 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 19:39:23.128039 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":246,"threads_started":1}
I20260926 19:39:23.130399 5601 heartbeater.cc:347] Connected to a master server at 127.2.164.126:38933
I20260926 19:39:23.130652 5601 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:23.131291 5601 heartbeater.cc:511] Master 127.2.164.126:38933 requested a full tablet report, sending...
I20260926 19:39:23.132575 5146 ts_manager.cc:194] Registered new tserver with Master: f97ae041294846e7bbff85ea112b4eac (127.2.164.67:44763)
I20260926 19:39:23.133349 5146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:36747
I20260926 19:39:23.141737 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 19:39:23.166831 5184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:39:23.164538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.2.164.126:38933)
0926 19:39:23.164702 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:23.164707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:23.165140 (+ 433us) server_negotiation.cc:407] Connection header received
0926 19:39:23.165314 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:23.165318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:23.165394 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:23.165477 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:23.166565 (+ 1088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56080: BlockingRecv error: recv got EOF from 127.0.0.1:56080 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:39:23.170251 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5207
W20260926 19:39:23.172531 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5207: Not authorized: Client connection negotiation failed: client connection to 127.2.164.65:34543: server requires authentication, but client does not have Kerberos credentials available
I20260926 19:39:23.172662 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 5207
I20260926 19:39:23.184911 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5340
W20260926 19:39:23.187206 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5340: Not authorized: Client connection negotiation failed: client connection to 127.2.164.66:43859: server requires authentication, but client does not have Kerberos credentials available
I20260926 19:39:23.187350 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 5340
I20260926 19:39:23.198788 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5473
W20260926 19:39:23.201428 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5473: Not authorized: Client connection negotiation failed: client connection to 127.2.164.67:44763: server requires authentication, but client does not have Kerberos credentials available
I20260926 19:39:23.201537 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 5473
I20260926 19:39:23.213125 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5114
I20260926 19:39:23.214145 5176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:23.313030 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 5114
2026-09-26T19:39:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3172 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:23 dist-test-slave-mrw4 krb5kdc[5618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:23 dist-test-slave-mrw4 krb5kdc[5618](info): set up 2 sockets
Sep 26 19:39:23 dist-test-slave-mrw4 krb5kdc[5618](info): commencing operation
krb5kdc: starting...
W20260926 19:39:24.297462 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:24 dist-test-slave-mrw4 krb5kdc[5618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:39:24.463891 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:37571
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:37571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:24.759821 5631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:24.760090 5631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:24.760213 5631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:24.767071 5631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:24.767174 5631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:24.767253 5631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:24.767326 5631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:24.767390 5631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:24.777643 5631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:37571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:37571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:24.778813 5631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:24.780047 5631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:24.790659 5636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:24.790707 5639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:24.790707 5637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:24.791680 5631 server_base.cc:1034] running on GCE node
I20260926 19:39:24.792311 5631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:24.793586 5631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:24.794775 5631 hybrid_clock.cc:648] HybridClock initialized: now 1790451564794731 us; error 40 us; skew 500 ppm
I20260926 19:39:24.795069 5631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:24 dist-test-slave-mrw4 krb5kdc[5618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451564, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:24.797961 5631 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:39:24.799289 5631 webserver.cc:460] Webserver started at http://127.2.164.126:45021/ using document root <none> and password file <none>
I20260926 19:39:24.799883 5631 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:24.799979 5631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:24.800247 5631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:24.802958 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "c5d058f015ec4e26a70da04018b8cfa0"
format_stamp: "Formatted at 2026-09-26 19:39:24 on dist-test-slave-mrw4"
server_key: "8d290d2e97efbc4fbdd8e2908d86381d"
server_key_iv: "842b05832806eaa3a8ffdca29385a8f1"
server_key_version: "encryptionkey@0"
I20260926 19:39:24.803659 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "c5d058f015ec4e26a70da04018b8cfa0"
format_stamp: "Formatted at 2026-09-26 19:39:24 on dist-test-slave-mrw4"
server_key: "8d290d2e97efbc4fbdd8e2908d86381d"
server_key_iv: "842b05832806eaa3a8ffdca29385a8f1"
server_key_version: "encryptionkey@0"
I20260926 19:39:24.808297 5631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 19:39:24.811362 5646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:24.812531 5631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:24.812671 5631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "c5d058f015ec4e26a70da04018b8cfa0"
format_stamp: "Formatted at 2026-09-26 19:39:24 on dist-test-slave-mrw4"
server_key: "8d290d2e97efbc4fbdd8e2908d86381d"
server_key_iv: "842b05832806eaa3a8ffdca29385a8f1"
server_key_version: "encryptionkey@0"
I20260926 19:39:24.812835 5631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:24.844544 5631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:24.848543 5631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:24.848784 5631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:24.865288 5631 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:37571
I20260926 19:39:24.865348 5697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:37571 every 8 connection(s)
I20260926 19:39:24.866540 5631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:24.868593 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 5631
I20260926 19:39:24.868796 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:24.869568 2705 external_mini_cluster.cc:1428] Setting key a7032704bdc5966597f2c8baa7ac1237
I20260926 19:39:24.870645 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:39:24 dist-test-slave-mrw4 krb5kdc[5618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:24.883978 5698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0: Bootstrap starting.
I20260926 19:39:24.886922 5698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:24.888051 5698 log.cc:824] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:24.891031 5698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0: No bootstrap required, opened a new log
I20260926 19:39:24.891158 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:24.871299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49720 (local address 127.2.164.126:37571)
0926 19:39:24.872014 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:24.872025 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:24.872062 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:39:24.873550 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:24.873571 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:24.873938 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:24.874263 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:24.875924 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:24.877437 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:24.879130 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:24.879770 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:24.879911 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:24.884276 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:24.884283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:24.884296 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:24.884332 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:24.887044 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:24.887788 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:24.887798 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:24.887804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:24.887955 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:24.888314 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:24.888317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:24.888319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:24.888844 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:24.889060 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:24.889547 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 19:39:24.889935 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":316,"threads_started":1}
I20260926 19:39:24.894831 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:39:24.898736 5698 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d058f015ec4e26a70da04018b8cfa0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37571 } }
I20260926 19:39:24.899031 5698 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:24.899118 5698 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d058f015ec4e26a70da04018b8cfa0, State: Initialized, Role: FOLLOWER
I20260926 19:39:24.899660 5698 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d058f015ec4e26a70da04018b8cfa0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37571 } }
I20260926 19:39:24.899863 5698 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:24.899981 5698 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:24.900122 5698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:24.902100 5698 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d058f015ec4e26a70da04018b8cfa0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37571 } }
I20260926 19:39:24.902524 5698 leader_election.cc:302] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5d058f015ec4e26a70da04018b8cfa0; no voters:
I20260926 19:39:24.903049 5698 leader_election.cc:288] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:24.903185 5704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:24.904152 5704 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [term 1 LEADER]: Becoming Leader. State: Replica: c5d058f015ec4e26a70da04018b8cfa0, State: Running, Role: LEADER
I20260926 19:39:24.904701 5704 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d058f015ec4e26a70da04018b8cfa0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37571 } }
I20260926 19:39:24.905104 5698 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:24.908149 5706 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5d058f015ec4e26a70da04018b8cfa0. Latest consensus state: current_term: 1 leader_uuid: "c5d058f015ec4e26a70da04018b8cfa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d058f015ec4e26a70da04018b8cfa0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37571 } } }
I20260926 19:39:24.908494 5706 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:24.909015 5705 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5d058f015ec4e26a70da04018b8cfa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d058f015ec4e26a70da04018b8cfa0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37571 } } }
I20260926 19:39:24.909411 5705 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:24.911557 5712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:24.916787 5712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:24.940795 5712 catalog_manager.cc:1347] Generated new cluster ID: 77a4a5267c6a4cbfaf936a79899e52b8
I20260926 19:39:24.940898 5712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:24.958233 5712 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:24.959565 5712 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:24.966432 5712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5d058f015ec4e26a70da04018b8cfa0: Generated new TSK 0
I20260926 19:39:24.967273 5712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:39:24.970275 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:24.960133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.2.164.126:37571)
0926 19:39:24.960312 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:24.960322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:24.962729 (+ 2407us) server_negotiation.cc:407] Connection header received
0926 19:39:24.962928 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:24.962933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:24.963007 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:24.963108 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:24.964349 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:24.965110 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:24.966828 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:24.967475 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:24.967615 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:24.968679 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:24.968685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:24.968690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:24.969530 (+ 840us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:24.969585 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:24.969618 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:24.969731 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:24.969844 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 19:39:24.969935 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:39:24.976951 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5631
I20260926 19:39:24.978092 5693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:25.064890 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 5631
2026-09-26T19:39:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1747 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:25 dist-test-slave-mrw4 krb5kdc[5733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:25 dist-test-slave-mrw4 krb5kdc[5733](info): set up 2 sockets
Sep 26 19:39:25 dist-test-slave-mrw4 krb5kdc[5733](info): commencing operation
krb5kdc: starting...
W20260926 19:39:26.049222 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:26 dist-test-slave-mrw4 krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:39:26.212378 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35137
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:35137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:26.509531 5746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:26.509800 5746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:26.509923 5746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:26.517076 5746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:26.517195 5746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:26.517328 5746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:26.517398 5746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:26.517465 5746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:26.526855 5746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:35137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:26.528069 5746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:26.529351 5746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:26.540086 5751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:26.540062 5752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:26.540017 5754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:26.540755 5746 server_base.cc:1034] running on GCE node
I20260926 19:39:26.541451 5746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:26.542675 5746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:26.543864 5746 hybrid_clock.cc:648] HybridClock initialized: now 1790451566543811 us; error 43 us; skew 500 ppm
I20260926 19:39:26.544170 5746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:26 dist-test-slave-mrw4 krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451566, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:26.546892 5746 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:39:26.548130 5746 webserver.cc:460] Webserver started at http://127.2.164.126:42239/ using document root <none> and password file <none>
I20260926 19:39:26.548702 5746 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:26.548812 5746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:26.549072 5746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:26.551729 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "0e7253dafcad4ba7a6095cd5cee49a6e"
format_stamp: "Formatted at 2026-09-26 19:39:26 on dist-test-slave-mrw4"
server_key: "b5a742280c511a9aa7e442d8d9ef7d16"
server_key_iv: "017ac8eda221ac2859b6a5ccdf2bd95a"
server_key_version: "encryptionkey@0"
I20260926 19:39:26.552407 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "0e7253dafcad4ba7a6095cd5cee49a6e"
format_stamp: "Formatted at 2026-09-26 19:39:26 on dist-test-slave-mrw4"
server_key: "b5a742280c511a9aa7e442d8d9ef7d16"
server_key_iv: "017ac8eda221ac2859b6a5ccdf2bd95a"
server_key_version: "encryptionkey@0"
I20260926 19:39:26.556941 5746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 19:39:26.559878 5761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:26.561026 5746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:26.561188 5746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "0e7253dafcad4ba7a6095cd5cee49a6e"
format_stamp: "Formatted at 2026-09-26 19:39:26 on dist-test-slave-mrw4"
server_key: "b5a742280c511a9aa7e442d8d9ef7d16"
server_key_iv: "017ac8eda221ac2859b6a5ccdf2bd95a"
server_key_version: "encryptionkey@0"
I20260926 19:39:26.561347 5746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:26.575133 5746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:26.578832 5746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:26.579080 5746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:26.595043 5746 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:35137
I20260926 19:39:26.595060 5812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:35137 every 8 connection(s)
I20260926 19:39:26.596288 5746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:26.597008 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 5746
I20260926 19:39:26.597221 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:26.597885 2705 external_mini_cluster.cc:1428] Setting key 9f8d6802267b30b08dce68f2f3c5573c
I20260926 19:39:26.600201 5813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:26.610025 5813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e: Bootstrap starting.
Sep 26 19:39:26 dist-test-slave-mrw4 krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:26.612986 5813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:26.613981 5813 log.cc:824] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:26.616947 5813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e: No bootstrap required, opened a new log
I20260926 19:39:26.618196 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:26.599683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.2.164.126:35137)
0926 19:39:26.600446 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:26.600460 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:26.600500 (+ 40us) server_negotiation.cc:407] Connection header received
0926 19:39:26.602037 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:26.602056 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:26.602423 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:26.602765 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:26.604052 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:26.605384 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:26.606553 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:26.607211 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:26.607395 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:26.611193 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:26.611203 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:26.611216 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:26.611254 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:26.614072 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:26.614794 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:26.614800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:26.614806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:26.614988 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:26.615370 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:26.615375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:26.615378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:26.615985 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:26.616272 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:26.616561 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 19:39:26.616959 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":303,"threads_started":1}
I20260926 19:39:26.624632 5813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7253dafcad4ba7a6095cd5cee49a6e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35137 } }
I20260926 19:39:26.624927 5813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:26.624992 5813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7253dafcad4ba7a6095cd5cee49a6e, State: Initialized, Role: FOLLOWER
I20260926 19:39:26.625514 5813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7253dafcad4ba7a6095cd5cee49a6e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35137 } }
I20260926 19:39:26.625752 5813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:26.625873 5813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:26.626022 5813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:26.627971 5813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7253dafcad4ba7a6095cd5cee49a6e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35137 } }
I20260926 19:39:26.628367 5813 leader_election.cc:302] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e7253dafcad4ba7a6095cd5cee49a6e; no voters:
I20260926 19:39:26.628820 5813 leader_election.cc:288] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:26.628935 5818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:26.629588 5818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [term 1 LEADER]: Becoming Leader. State: Replica: 0e7253dafcad4ba7a6095cd5cee49a6e, State: Running, Role: LEADER
I20260926 19:39:26.630080 5818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7253dafcad4ba7a6095cd5cee49a6e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35137 } }
I20260926 19:39:26.630615 5813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:26.632495 5819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e7253dafcad4ba7a6095cd5cee49a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7253dafcad4ba7a6095cd5cee49a6e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35137 } } }
I20260926 19:39:26.632823 5819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:26.633397 5820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e7253dafcad4ba7a6095cd5cee49a6e. Latest consensus state: current_term: 1 leader_uuid: "0e7253dafcad4ba7a6095cd5cee49a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7253dafcad4ba7a6095cd5cee49a6e" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35137 } } }
I20260926 19:39:26.633770 5820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:26.634903 5825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:26.639492 5825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:26.647488 5825 catalog_manager.cc:1347] Generated new cluster ID: a8c90937c10e44309b8de508af6fae0a
I20260926 19:39:26.647583 5825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:26.659154 5825 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:26.660815 5825 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:26.669102 5825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e7253dafcad4ba7a6095cd5cee49a6e: Generated new TSK 0
I20260926 19:39:26.669946 5825 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:39:26.714272 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35137
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:27.010380 5839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:27.010625 5839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:27.010740 5839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:27.017611 5839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:27.017707 5839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:27.017854 5839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:27.026857 5839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:27.028029 5839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:27.029253 5839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:27.041231 5844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:27.041263 5845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:27.041285 5847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:27.042116 5839 server_base.cc:1034] running on GCE node
I20260926 19:39:27.042654 5839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:27.043479 5839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:27.044659 5839 hybrid_clock.cc:648] HybridClock initialized: now 1790451567044617 us; error 33 us; skew 500 ppm
I20260926 19:39:27.044938 5839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:27 dist-test-slave-mrw4 krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451567, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:27.047755 5839 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:39:27.049099 5839 webserver.cc:460] Webserver started at http://127.2.164.65:40569/ using document root <none> and password file <none>
I20260926 19:39:27.049741 5839 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:27.049839 5839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:27.050112 5839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:27.052762 5839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "38265e3a5672497c8d631833d9681bc8"
format_stamp: "Formatted at 2026-09-26 19:39:27 on dist-test-slave-mrw4"
server_key: "6840f08bafebc8697fa769cf665c7332"
server_key_iv: "e61980467cd22927a386a2cfa4843373"
server_key_version: "encryptionkey@0"
I20260926 19:39:27.053505 5839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "38265e3a5672497c8d631833d9681bc8"
format_stamp: "Formatted at 2026-09-26 19:39:27 on dist-test-slave-mrw4"
server_key: "6840f08bafebc8697fa769cf665c7332"
server_key_iv: "e61980467cd22927a386a2cfa4843373"
server_key_version: "encryptionkey@0"
I20260926 19:39:27.058149 5839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:39:27.061313 5854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:27.062608 5839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:39:27.062744 5839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "38265e3a5672497c8d631833d9681bc8"
format_stamp: "Formatted at 2026-09-26 19:39:27 on dist-test-slave-mrw4"
server_key: "6840f08bafebc8697fa769cf665c7332"
server_key_iv: "e61980467cd22927a386a2cfa4843373"
server_key_version: "encryptionkey@0"
I20260926 19:39:27.062911 5839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:27.077440 5839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:27.080857 5839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:27.081099 5839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:27.081984 5839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:27.083192 5839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:27.083259 5839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:27.083343 5839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:27.083390 5839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:27.111230 5839 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:34797
I20260926 19:39:27.111269 5966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:34797 every 8 connection(s)
I20260926 19:39:27.112396 5839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:27.118222 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 5839
I20260926 19:39:27.118389 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:27.118988 2705 external_mini_cluster.cc:1428] Setting key 426adaa185c1e243558d43e54c765918
Sep 26 19:39:27 dist-test-slave-mrw4 krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451567, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:27.131574 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:27.114789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:40157 (local address 127.2.164.126:35137)
0926 19:39:27.114961 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:27.114966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:27.116039 (+ 1073us) server_negotiation.cc:407] Connection header received
0926 19:39:27.117902 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:27.117908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:27.117993 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:27.118096 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:27.119480 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:27.120191 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:27.121822 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:27.122419 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:27.122539 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:27.125677 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:27.125683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:27.125688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:27.125726 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:27.128379 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:27.129250 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:27.129254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:27.129256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:27.129382 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:27.129888 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:27.129892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:27.129894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:27.130184 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:27.130310 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:27.131113 (+ 803us) server_negotiation.cc:299] Negotiation successful
0926 19:39:27.131287 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:39:27.132851 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:27.115236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35137 (local address 127.2.164.65:40157)
0926 19:39:27.115893 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:27.115934 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:27.117632 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:27.118247 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:27.118255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:27.118754 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:27.119322 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:27.119340 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:27.120337 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:27.120342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:27.121682 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:27.121694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:27.122586 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:27.122591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:27.122772 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:27.123351 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:27.123372 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:27.125473 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:27.128526 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:27.128535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:27.128551 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:27.129079 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:27.129473 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:27.129478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:27.129481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:27.129709 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:27.130322 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:27.130330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:27.130724 (+ 394us) client_negotiation.cc:769] Sending connection context
0926 19:39:27.130982 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 19:39:27.131513 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":237,"threads_started":1}
I20260926 19:39:27.134353 5967 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35137
I20260926 19:39:27.134645 5967 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260926 19:39:27.135354 5967 heartbeater.cc:511] Master 127.2.164.126:35137 requested a full tablet report, sending...
I20260926 19:39:27.137553 5778 ts_manager.cc:194] Registered new tserver with Master: 38265e3a5672497c8d631833d9681bc8 (127.2.164.65:34797)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260926 19:39:27.139444 5778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:40157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:39:27.159817 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35137
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:27.456892 5972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:27.457209 5972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:27.457358 5972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:27.464926 5972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:27.465025 5972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:27.465217 5972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:27.474571 5972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:27.475720 5972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:27.476902 5972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:27.488680 5980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:27.488750 5977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:27.488729 5978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:27.489463 5972 server_base.cc:1034] running on GCE node
I20260926 19:39:27.489975 5972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:27.490826 5972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:27.492022 5972 hybrid_clock.cc:648] HybridClock initialized: now 1790451567491973 us; error 38 us; skew 500 ppm
I20260926 19:39:27.492298 5972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:27 dist-test-slave-mrw4 krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451567, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:27.495127 5972 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:39:27.496385 5972 webserver.cc:460] Webserver started at http://127.2.164.66:32923/ using document root <none> and password file <none>
I20260926 19:39:27.496984 5972 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:27.497092 5972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:27.497382 5972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:27.500015 5972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "cd28ac12086045be8fd7c6c3b1e50800"
format_stamp: "Formatted at 2026-09-26 19:39:27 on dist-test-slave-mrw4"
server_key: "88b555eb85157ec52149730e173298c7"
server_key_iv: "b625f57f8b31a5c7e235f07650aa5294"
server_key_version: "encryptionkey@0"
I20260926 19:39:27.500720 5972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "cd28ac12086045be8fd7c6c3b1e50800"
format_stamp: "Formatted at 2026-09-26 19:39:27 on dist-test-slave-mrw4"
server_key: "88b555eb85157ec52149730e173298c7"
server_key_iv: "b625f57f8b31a5c7e235f07650aa5294"
server_key_version: "encryptionkey@0"
I20260926 19:39:27.505460 5972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 19:39:27.508634 5987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:27.509920 5972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:39:27.510058 5972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "cd28ac12086045be8fd7c6c3b1e50800"
format_stamp: "Formatted at 2026-09-26 19:39:27 on dist-test-slave-mrw4"
server_key: "88b555eb85157ec52149730e173298c7"
server_key_iv: "b625f57f8b31a5c7e235f07650aa5294"
server_key_version: "encryptionkey@0"
I20260926 19:39:27.510216 5972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:27.522789 5972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:27.526719 5972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:27.526983 5972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:27.527850 5972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:27.529131 5972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:27.529242 5972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:27.529336 5972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:27.529382 5972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:27.558362 5972 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:42919
I20260926 19:39:27.558403 6099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:42919 every 8 connection(s)
I20260926 19:39:27.559548 5972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:27.564419 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 5972
I20260926 19:39:27.564577 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:27.565097 2705 external_mini_cluster.cc:1428] Setting key a29f7fc1af3f54ef0b6359243d18b2ed
Sep 26 19:39:27 dist-test-slave-mrw4 krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451567, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:27.579957 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:27.561943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:39943 (local address 127.2.164.126:35137)
0926 19:39:27.562143 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:27.562148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:27.563259 (+ 1111us) server_negotiation.cc:407] Connection header received
0926 19:39:27.565404 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:27.565408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:27.565496 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:27.565583 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:27.567308 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:27.567987 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:27.570048 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:27.570447 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:27.570541 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:27.574214 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:27.574220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:27.574225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:27.574261 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:27.577011 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:27.577697 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:27.577702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:27.577705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:27.577842 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:27.578243 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:27.578247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:27.578250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:27.578565 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:27.578714 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:27.579396 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 19:39:27.579609 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 19:39:27.580698 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:27.562448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35137 (local address 127.2.164.66:39943)
0926 19:39:27.563105 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:27.563142 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:27.565102 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:27.565853 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:27.565861 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:27.566418 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:27.567117 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:27.567143 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:27.568136 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:27.568142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:27.569920 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:27.569936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:27.570639 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:27.570646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:27.570921 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:27.571560 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:27.571586 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:27.574031 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:27.577142 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:27.577149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:27.577161 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:27.577586 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:27.577950 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:27.577953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:27.577956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:27.578139 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:27.578686 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:27.578693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:27.578993 (+ 300us) client_negotiation.cc:769] Sending connection context
0926 19:39:27.579272 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 19:39:27.579636 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":260,"threads_started":1}
I20260926 19:39:27.581982 6100 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35137
I20260926 19:39:27.582248 6100 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260926 19:39:27.582991 6100 heartbeater.cc:511] Master 127.2.164.126:35137 requested a full tablet report, sending...
I20260926 19:39:27.584266 5778 ts_manager.cc:194] Registered new tserver with Master: cd28ac12086045be8fd7c6c3b1e50800 (127.2.164.66:42919)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
I20260926 19:39:27.585139 5778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:39943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:39:27.606328 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35137
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:27.901034 6105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:27.901298 6105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:27.901392 6105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:27.908227 6105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:27.908324 6105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:27.908428 6105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:27.917680 6105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:27.918792 6105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:27.919996 6105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:27.931972 6113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:27.932011 6111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:27.932036 6110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:27.932641 6105 server_base.cc:1034] running on GCE node
I20260926 19:39:27.933190 6105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:27.933986 6105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:27.935173 6105 hybrid_clock.cc:648] HybridClock initialized: now 1790451567935114 us; error 47 us; skew 500 ppm
I20260926 19:39:27.935443 6105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:27 dist-test-slave-mrw4 krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451567, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:27.938295 6105 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:39:27.939580 6105 webserver.cc:460] Webserver started at http://127.2.164.67:44465/ using document root <none> and password file <none>
I20260926 19:39:27.940168 6105 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:27.940274 6105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:27.940524 6105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:27.943151 6105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "38b0921f03754391871eaa154ca01cf4"
format_stamp: "Formatted at 2026-09-26 19:39:27 on dist-test-slave-mrw4"
server_key: "fd69dde2109cd37c3c6908db5bf59ccf"
server_key_iv: "ce27dc476c30943437036b921c49e220"
server_key_version: "encryptionkey@0"
I20260926 19:39:27.943856 6105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "38b0921f03754391871eaa154ca01cf4"
format_stamp: "Formatted at 2026-09-26 19:39:27 on dist-test-slave-mrw4"
server_key: "fd69dde2109cd37c3c6908db5bf59ccf"
server_key_iv: "ce27dc476c30943437036b921c49e220"
server_key_version: "encryptionkey@0"
I20260926 19:39:27.948417 6105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 19:39:27.951572 6120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:27.952729 6105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:27.952863 6105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "38b0921f03754391871eaa154ca01cf4"
format_stamp: "Formatted at 2026-09-26 19:39:27 on dist-test-slave-mrw4"
server_key: "fd69dde2109cd37c3c6908db5bf59ccf"
server_key_iv: "ce27dc476c30943437036b921c49e220"
server_key_version: "encryptionkey@0"
I20260926 19:39:27.953034 6105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:27.970005 6105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:27.973469 6105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:27.973734 6105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:27.974563 6105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:27.975754 6105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:27.975826 6105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:27.975941 6105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:27.975991 6105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:28.004247 6105 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:43731
I20260926 19:39:28.004289 6232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:43731 every 8 connection(s)
I20260926 19:39:28.005497 6105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:28.010408 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 6105
I20260926 19:39:28.010566 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:28.011094 2705 external_mini_cluster.cc:1428] Setting key d743f7c83ab6f956164322f171dfb6e5
Sep 26 19:39:28 dist-test-slave-mrw4 krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451567, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:28.023980 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:28.007855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:49841 (local address 127.2.164.126:35137)
0926 19:39:28.008058 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:28.008065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:28.009188 (+ 1123us) server_negotiation.cc:407] Connection header received
0926 19:39:28.011300 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:28.011305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:28.011371 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:28.011455 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:28.013361 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.014310 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.015851 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.016239 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.016339 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:28.019323 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:28.019327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:28.019331 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:28.019359 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:28.021738 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:28.022319 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:28.022323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:28.022326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:28.022454 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:28.022761 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:28.022764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:28.022765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:28.022999 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:28.023099 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:28.023633 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 19:39:28.023778 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:39:28.025019 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:28.008359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35137 (local address 127.2.164.67:49841)
0926 19:39:28.009025 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:28.009068 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:28.010998 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:28.011699 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:28.011707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:28.012384 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:28.013143 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:28.013186 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.014473 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.014479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:28.015711 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:28.015723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.016410 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.016415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:28.016589 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:28.017145 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:28.017194 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:28.019147 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:28.021847 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:28.021853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:28.021864 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:28.022223 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:28.022531 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:28.022534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:28.022536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:28.022686 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:28.023096 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:28.023102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:28.023370 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 19:39:28.023569 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 19:39:28.023897 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":245,"threads_started":1}
I20260926 19:39:28.026328 6233 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35137
I20260926 19:39:28.026582 6233 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:28.027243 6233 heartbeater.cc:511] Master 127.2.164.126:35137 requested a full tablet report, sending...
I20260926 19:39:28.028443 5777 ts_manager.cc:194] Registered new tserver with Master: 38b0921f03754391871eaa154ca01cf4 (127.2.164.67:43731)
I20260926 19:39:28.029253 5777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:49841
I20260926 19:39:28.037964 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:39:28 dist-test-slave-mrw4 krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451568, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 19:39:28 dist-test-slave-mrw4 krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451568, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:28.085285 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:28.066966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.2.164.126:35137)
0926 19:39:28.067174 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:28.067180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:28.067513 (+ 333us) server_negotiation.cc:407] Connection header received
0926 19:39:28.067684 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:28.067690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:28.067772 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:28.067854 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:28.069148 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.069887 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.071066 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.071484 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.071591 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:28.080599 (+ 9008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:28.080605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:28.080608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:28.080643 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:28.082885 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:28.083458 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:28.083462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:28.083465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:28.083577 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:28.083927 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:28.083932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:28.083935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:28.084224 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:28.084459 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:28.084714 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 19:39:28.084919 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 19:39:28.085811 5777 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40322
I20260926 19:39:28.087285 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5839
Sep 26 19:39:28 dist-test-slave-mrw4 krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451568, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:28.101364 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:28.088076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.2.164.65:34797)
0926 19:39:28.088633 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:28.088640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:28.088661 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:39:28.088712 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:28.088718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:28.088938 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:28.089077 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:28.090146 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.091091 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.092224 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.092748 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.092854 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:28.096201 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:28.096211 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:28.096225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:28.096255 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:28.098842 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:28.099436 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:28.099442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:28.099445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:28.099557 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:28.099884 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:28.099889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:28.099892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:28.100214 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:28.100485 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:28.100751 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 19:39:28.101004 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":280,"threads_started":1}
W20260926 19:39:28.102546 5962 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51364
W20260926 19:39:28.103546 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5839: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:39:28.103732 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 5839
I20260926 19:39:28.115892 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5972
Sep 26 19:39:28 dist-test-slave-mrw4 krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451568, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:39:28.129855 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:28.116594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.2.164.66:42919)
0926 19:39:28.117053 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:28.117060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:28.117083 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:39:28.117135 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:28.117141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:28.117380 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:28.117507 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:28.118651 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.119588 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.120781 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.121372 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.121467 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:28.125051 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:28.125062 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:28.125074 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:28.125103 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:28.127585 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:28.128117 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:28.128123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:28.128125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:28.128231 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:28.128557 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:28.128561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:28.128563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:28.128885 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:28.129107 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:28.129334 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 19:39:28.129520 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":216,"threads_started":1}
W20260926 19:39:28.131081 6095 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36878
W20260926 19:39:28.132091 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5972: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:39:28.132221 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 5972
I20260926 19:39:28.144091 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 6105
Sep 26 19:39:28 dist-test-slave-mrw4 krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451568, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:39:28.158236 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:28.144793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.2.164.67:43731)
0926 19:39:28.145376 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:28.145405 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:28.145432 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:39:28.145486 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:28.145492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:28.145685 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:28.145806 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:28.146950 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.147978 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.149132 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:28.149770 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:28.149886 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:28.153554 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:28.153564 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:28.153577 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:28.153606 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:28.155990 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:28.156561 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:28.156568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:28.156570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:28.156680 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:28.156962 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:28.156966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:28.156968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:28.157371 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:28.157623 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:28.157789 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 19:39:28.157966 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":275,"threads_started":1}
W20260926 19:39:28.159409 6228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45076
W20260926 19:39:28.160375 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 6105: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:39:28.160521 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 6105
I20260926 19:39:28.172086 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 5746
I20260926 19:39:28.172994 5808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:28.273629 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 5746
2026-09-26T19:39:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3213 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:28 dist-test-slave-mrw4 krb5kdc[6250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:28 dist-test-slave-mrw4 krb5kdc[6250](info): set up 2 sockets
Sep 26 19:39:28 dist-test-slave-mrw4 krb5kdc[6250](info): commencing operation
krb5kdc: starting...
W20260926 19:39:29.265198 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:29 dist-test-slave-mrw4 krb5kdc[6250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:39:29.436051 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33167
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:33167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:29.730993 6263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:29.731282 6263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:29.731406 6263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:29.738323 6263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:29.738437 6263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:29.738515 6263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:29.738590 6263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:29.738648 6263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:29.747967 6263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:33167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:29.749296 6263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:29.750522 6263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:29.761104 6268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:29.761143 6271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:29.761204 6269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:29.761921 6263 server_base.cc:1034] running on GCE node
I20260926 19:39:29.762630 6263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:29.763845 6263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:29.765036 6263 hybrid_clock.cc:648] HybridClock initialized: now 1790451569764995 us; error 27 us; skew 500 ppm
I20260926 19:39:29.765359 6263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:29 dist-test-slave-mrw4 krb5kdc[6250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451569, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:29.768160 6263 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:39:29.769507 6263 webserver.cc:460] Webserver started at http://127.2.164.126:42541/ using document root <none> and password file <none>
I20260926 19:39:29.770104 6263 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:29.770215 6263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:29.770493 6263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:29.773206 6263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "ae39679ab4de4aac8d8d12cd3cf77964"
format_stamp: "Formatted at 2026-09-26 19:39:29 on dist-test-slave-mrw4"
server_key: "aa34d7485e55cc1f62774f4fc1a7a762"
server_key_iv: "75c8f620d3eecef5b4bedc7979cfbd5b"
server_key_version: "encryptionkey@0"
I20260926 19:39:29.773917 6263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "ae39679ab4de4aac8d8d12cd3cf77964"
format_stamp: "Formatted at 2026-09-26 19:39:29 on dist-test-slave-mrw4"
server_key: "aa34d7485e55cc1f62774f4fc1a7a762"
server_key_iv: "75c8f620d3eecef5b4bedc7979cfbd5b"
server_key_version: "encryptionkey@0"
I20260926 19:39:29.778470 6263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 19:39:29.781519 6278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:29.782672 6263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260926 19:39:29.782815 6263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "ae39679ab4de4aac8d8d12cd3cf77964"
format_stamp: "Formatted at 2026-09-26 19:39:29 on dist-test-slave-mrw4"
server_key: "aa34d7485e55cc1f62774f4fc1a7a762"
server_key_iv: "75c8f620d3eecef5b4bedc7979cfbd5b"
server_key_version: "encryptionkey@0"
I20260926 19:39:29.782981 6263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:29.797734 6263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:29.801577 6263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:29.801811 6263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:29.817612 6263 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:33167
I20260926 19:39:29.817657 6329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:33167 every 8 connection(s)
I20260926 19:39:29.818822 6263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:29.820210 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 6263
I20260926 19:39:29.820417 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:29.821185 2705 external_mini_cluster.cc:1428] Setting key 801efd62747fe635485d6565eb8d8d48
I20260926 19:39:29.823114 6330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:39:29 dist-test-slave-mrw4 krb5kdc[6250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:29.835713 6330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964: Bootstrap starting.
I20260926 19:39:29.838929 6330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:29.840071 6330 log.cc:824] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:29.843086 6330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964: No bootstrap required, opened a new log
I20260926 19:39:29.843384 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:29.823089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.2.164.126:33167)
0926 19:39:29.823816 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:29.823830 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:29.823876 (+ 46us) server_negotiation.cc:407] Connection header received
0926 19:39:29.825908 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:29.825934 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:29.826406 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:29.826822 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:29.828042 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:29.830021 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:29.831243 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:29.832193 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:29.832377 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:29.836272 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:29.836280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:29.836293 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:29.836325 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:29.839086 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:29.839827 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:29.839833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:29.839837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:29.840001 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:29.840371 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:29.840374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:29.840376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:29.840844 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:29.841071 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:29.841622 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 19:39:29.842159 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":303,"threads_started":1}
I20260926 19:39:29.850965 6330 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae39679ab4de4aac8d8d12cd3cf77964" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33167 } }
I20260926 19:39:29.851264 6330 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:29.851331 6330 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae39679ab4de4aac8d8d12cd3cf77964, State: Initialized, Role: FOLLOWER
I20260926 19:39:29.851771 6330 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae39679ab4de4aac8d8d12cd3cf77964" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33167 } }
I20260926 19:39:29.851960 6330 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:29.852046 6330 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:29.852161 6330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:29.854112 6330 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae39679ab4de4aac8d8d12cd3cf77964" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33167 } }
I20260926 19:39:29.854512 6330 leader_election.cc:302] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae39679ab4de4aac8d8d12cd3cf77964; no voters:
I20260926 19:39:29.855186 6330 leader_election.cc:288] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:29.855335 6335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:29.856011 6335 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [term 1 LEADER]: Becoming Leader. State: Replica: ae39679ab4de4aac8d8d12cd3cf77964, State: Running, Role: LEADER
I20260926 19:39:29.856458 6335 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae39679ab4de4aac8d8d12cd3cf77964" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33167 } }
I20260926 19:39:29.856978 6330 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:29.858805 6336 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae39679ab4de4aac8d8d12cd3cf77964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae39679ab4de4aac8d8d12cd3cf77964" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33167 } } }
I20260926 19:39:29.859158 6336 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:29.859566 6337 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae39679ab4de4aac8d8d12cd3cf77964. Latest consensus state: current_term: 1 leader_uuid: "ae39679ab4de4aac8d8d12cd3cf77964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae39679ab4de4aac8d8d12cd3cf77964" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33167 } } }
I20260926 19:39:29.859844 6337 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:29.859942 6344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:29.865728 6344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:29.877851 6344 catalog_manager.cc:1347] Generated new cluster ID: 8148e816bff44530be7e10ad41554e90
I20260926 19:39:29.877947 6344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:29.894011 6344 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:29.895147 6344 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:29.903988 6344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae39679ab4de4aac8d8d12cd3cf77964: Generated new TSK 0
I20260926 19:39:29.904685 6344 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:39:29.953320 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33167
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:30.257467 6356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:30.257758 6356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:30.257886 6356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:30.264878 6356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:30.264983 6356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:30.265132 6356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:30.274222 6356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:30.275336 6356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:30.276537 6356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:30.288527 6361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:30.288568 6362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:30.288599 6364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:30.289271 6356 server_base.cc:1034] running on GCE node
I20260926 19:39:30.289803 6356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:30.290642 6356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:30.291831 6356 hybrid_clock.cc:648] HybridClock initialized: now 1790451570291778 us; error 47 us; skew 500 ppm
I20260926 19:39:30.292112 6356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:30 dist-test-slave-mrw4 krb5kdc[6250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451570, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:30.294942 6356 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:39:30.296234 6356 webserver.cc:460] Webserver started at http://127.2.164.65:40905/ using document root <none> and password file <none>
I20260926 19:39:30.296838 6356 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:30.296952 6356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:30.297257 6356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:30.299925 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "bbb3494c688042cfb7302a9af779cd05"
format_stamp: "Formatted at 2026-09-26 19:39:30 on dist-test-slave-mrw4"
server_key: "8434a5ceab0330336002e225adc6cb1a"
server_key_iv: "f42b1988c7d718f656e97efbf4d879f4"
server_key_version: "encryptionkey@0"
I20260926 19:39:30.300640 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "bbb3494c688042cfb7302a9af779cd05"
format_stamp: "Formatted at 2026-09-26 19:39:30 on dist-test-slave-mrw4"
server_key: "8434a5ceab0330336002e225adc6cb1a"
server_key_iv: "f42b1988c7d718f656e97efbf4d879f4"
server_key_version: "encryptionkey@0"
I20260926 19:39:30.305538 6356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 19:39:30.308691 6371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:30.310070 6356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:39:30.310196 6356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "bbb3494c688042cfb7302a9af779cd05"
format_stamp: "Formatted at 2026-09-26 19:39:30 on dist-test-slave-mrw4"
server_key: "8434a5ceab0330336002e225adc6cb1a"
server_key_iv: "f42b1988c7d718f656e97efbf4d879f4"
server_key_version: "encryptionkey@0"
I20260926 19:39:30.310356 6356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:30.334175 6356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:30.338084 6356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:30.338302 6356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:30.339154 6356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:30.340361 6356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:30.340430 6356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:30.340518 6356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:30.340559 6356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:30.367874 6356 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:39155
I20260926 19:39:30.367934 6483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:39155 every 8 connection(s)
I20260926 19:39:30.369046 6356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:30.370397 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 6356
I20260926 19:39:30.370586 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:30.371234 2705 external_mini_cluster.cc:1428] Setting key ae1e8fe481291a194a28c80f87ece130
Sep 26 19:39:30 dist-test-slave-mrw4 krb5kdc[6250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451570, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260926 19:39:30.388048 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:30.371533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:46839 (local address 127.2.164.126:33167)
0926 19:39:30.371911 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:30.371916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:30.373098 (+ 1182us) server_negotiation.cc:407] Connection header received
0926 19:39:30.374855 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:30.374860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:30.374934 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:30.375025 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:30.376505 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:30.377185 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:30.378771 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:30.379189 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:30.379302 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:30.382587 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:30.382592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:30.382595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:30.382624 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:30.385082 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:30.385753 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:30.385757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:30.385759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:30.385885 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:30.386410 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:30.386414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:30.386416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:30.386660 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:30.386762 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:30.387618 (+ 856us) server_negotiation.cc:299] Negotiation successful
0926 19:39:30.387773 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":198,"threads_started":1}
I20260926 19:39:30.388953 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:30.372164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33167 (local address 127.2.164.65:46839)
0926 19:39:30.372906 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:30.372976 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:30.374607 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:30.375261 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:30.375268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:30.375735 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:30.376340 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:30.376366 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:30.377323 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:30.377327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:30.378638 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:30.378651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:30.379398 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:30.379405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:30.379618 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:30.380238 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:30.380259 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:30.382356 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:30.385231 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:30.385239 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:30.385253 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:30.385645 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:30.386037 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:30.386042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:30.386046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:30.386271 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:30.386785 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:30.386793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:30.387220 (+ 427us) client_negotiation.cc:769] Sending connection context
0926 19:39:30.387512 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 19:39:30.387870 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":270,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260926 19:39:30.390389 6484 heartbeater.cc:347] Connected to a master server at 127.2.164.126:33167
I20260926 19:39:30.390657 6484 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:30.391290 6484 heartbeater.cc:511] Master 127.2.164.126:33167 requested a full tablet report, sending...
I20260926 19:39:30.393291 6295 ts_manager.cc:194] Registered new tserver with Master: bbb3494c688042cfb7302a9af779cd05 (127.2.164.65:39155)
I20260926 19:39:30.394697 6295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:46839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:39:30.409909 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33167
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:30.706683 6490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:30.706954 6490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:30.707090 6490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:30.713973 6490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:30.714077 6490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:30.714219 6490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:30.723239 6490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:30.724359 6490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:30.725566 6490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:30.737205 6495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:30.737279 6496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:30.737308 6498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:30.738157 6490 server_base.cc:1034] running on GCE node
I20260926 19:39:30.738713 6490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:30.739535 6490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:30.740723 6490 hybrid_clock.cc:648] HybridClock initialized: now 1790451570740667 us; error 49 us; skew 500 ppm
I20260926 19:39:30.740991 6490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:30 dist-test-slave-mrw4 krb5kdc[6250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451570, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:30.743863 6490 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:39:30.745076 6490 webserver.cc:460] Webserver started at http://127.2.164.66:37401/ using document root <none> and password file <none>
I20260926 19:39:30.745698 6490 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:30.745792 6490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:30.746025 6490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:30.748621 6490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "50a8a687066e4f468f4358b1225a18b5"
format_stamp: "Formatted at 2026-09-26 19:39:30 on dist-test-slave-mrw4"
server_key: "dba7e08728667267aa8cc18ebed82017"
server_key_iv: "e8fc90a7efac195c429e05f44a4c7aed"
server_key_version: "encryptionkey@0"
I20260926 19:39:30.749349 6490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "50a8a687066e4f468f4358b1225a18b5"
format_stamp: "Formatted at 2026-09-26 19:39:30 on dist-test-slave-mrw4"
server_key: "dba7e08728667267aa8cc18ebed82017"
server_key_iv: "e8fc90a7efac195c429e05f44a4c7aed"
server_key_version: "encryptionkey@0"
I20260926 19:39:30.753893 6490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:39:30.756889 6505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:30.758142 6490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:39:30.758262 6490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "50a8a687066e4f468f4358b1225a18b5"
format_stamp: "Formatted at 2026-09-26 19:39:30 on dist-test-slave-mrw4"
server_key: "dba7e08728667267aa8cc18ebed82017"
server_key_iv: "e8fc90a7efac195c429e05f44a4c7aed"
server_key_version: "encryptionkey@0"
I20260926 19:39:30.758409 6490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:30.770243 6490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:30.773938 6490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:30.774173 6490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:30.774987 6490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:30.776144 6490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:30.776217 6490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:30.776326 6490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:30.776381 6490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:30.805017 6490 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:40017
I20260926 19:39:30.805058 6617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:40017 every 8 connection(s)
I20260926 19:39:30.806398 6490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:30.814764 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 6490
I20260926 19:39:30.814926 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:30.815294 2705 external_mini_cluster.cc:1428] Setting key f18dcaad024c584d80a6eba494f20a3d
Sep 26 19:39:30 dist-test-slave-mrw4 krb5kdc[6250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451570, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:30.826294 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:30.808631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:34671 (local address 127.2.164.126:33167)
0926 19:39:30.808796 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:30.808801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:30.809985 (+ 1184us) server_negotiation.cc:407] Connection header received
0926 19:39:30.811671 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:30.811675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:30.811738 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:30.811820 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:30.813221 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:30.813964 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:30.815505 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:30.815992 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:30.816116 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:30.820023 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:30.820030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:30.820035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:30.820070 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:30.823267 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:30.823881 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:30.823886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:30.823890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:30.824050 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:30.824541 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:30.824546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:30.824549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:30.824895 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:30.825032 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:30.825783 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 19:39:30.825990 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:39:30.827322 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:30.809145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33167 (local address 127.2.164.66:34671)
0926 19:39:30.809839 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:30.809875 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:30.811423 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:30.811999 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:30.812007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:30.812480 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:30.813056 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:30.813074 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:30.814087 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:30.814091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:30.815364 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:30.815378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:30.816175 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:30.816182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:30.816439 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:30.817129 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:30.817156 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:30.819815 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:30.823408 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:30.823415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:30.823425 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:30.823790 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:30.824166 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:30.824171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:30.824175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:30.824415 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:30.825041 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:30.825050 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:30.825441 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 19:39:30.825697 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 19:39:30.826157 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":259,"threads_started":1}
I20260926 19:39:30.828828 6618 heartbeater.cc:347] Connected to a master server at 127.2.164.126:33167
I20260926 19:39:30.829128 6618 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:30.829901 6618 heartbeater.cc:511] Master 127.2.164.126:33167 requested a full tablet report, sending...
I20260926 19:39:30.831472 6295 ts_manager.cc:194] Registered new tserver with Master: 50a8a687066e4f468f4358b1225a18b5 (127.2.164.66:40017)
I20260926 19:39:30.832600 6295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:34671
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:39:30.854503 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33167
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:31.152405 6623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:31.152653 6623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:31.152740 6623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:31.159889 6623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:31.159977 6623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:31.160090 6623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:31.169234 6623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:31.170262 6623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:31.171408 6623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:31.182870 6628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:31.182960 6631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:31.182987 6629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:31.183929 6623 server_base.cc:1034] running on GCE node
I20260926 19:39:31.184437 6623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:31.185237 6623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:31.186442 6623 hybrid_clock.cc:648] HybridClock initialized: now 1790451571186390 us; error 31 us; skew 500 ppm
I20260926 19:39:31.186705 6623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:31 dist-test-slave-mrw4 krb5kdc[6250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451571, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:31.189692 6623 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:39:31.191016 6623 webserver.cc:460] Webserver started at http://127.2.164.67:32835/ using document root <none> and password file <none>
I20260926 19:39:31.191630 6623 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:31.191747 6623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:31.192013 6623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:31.194733 6623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "9c15f859b4c543b187662c4ac69c34dc"
format_stamp: "Formatted at 2026-09-26 19:39:31 on dist-test-slave-mrw4"
server_key: "711294d16ee0916942f48f5683c0b638"
server_key_iv: "e6593b4e4d6609c069c47e188d95a53d"
server_key_version: "encryptionkey@0"
I20260926 19:39:31.195439 6623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "9c15f859b4c543b187662c4ac69c34dc"
format_stamp: "Formatted at 2026-09-26 19:39:31 on dist-test-slave-mrw4"
server_key: "711294d16ee0916942f48f5683c0b638"
server_key_iv: "e6593b4e4d6609c069c47e188d95a53d"
server_key_version: "encryptionkey@0"
I20260926 19:39:31.200162 6623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:39:31.203392 6638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:31.204628 6623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 19:39:31.204767 6623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "9c15f859b4c543b187662c4ac69c34dc"
format_stamp: "Formatted at 2026-09-26 19:39:31 on dist-test-slave-mrw4"
server_key: "711294d16ee0916942f48f5683c0b638"
server_key_iv: "e6593b4e4d6609c069c47e188d95a53d"
server_key_version: "encryptionkey@0"
I20260926 19:39:31.204937 6623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:31.224753 6623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:31.228482 6623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:31.228744 6623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:31.229631 6623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:31.230865 6623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:31.230939 6623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:31.231057 6623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:31.231109 6623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:31.259800 6623 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:33143
I20260926 19:39:31.259832 6750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:33143 every 8 connection(s)
I20260926 19:39:31.261046 6623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:31.269371 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 6623
I20260926 19:39:31.269539 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:31.269920 2705 external_mini_cluster.cc:1428] Setting key 5b38befb44cabb4368dea57ca9ea9c12
Sep 26 19:39:31 dist-test-slave-mrw4 krb5kdc[6250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451571, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:31.279091 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:31.263460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:57681 (local address 127.2.164.126:33167)
0926 19:39:31.263614 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:31.263619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:31.264761 (+ 1142us) server_negotiation.cc:407] Connection header received
0926 19:39:31.266517 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:31.266521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:31.266586 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:31.266671 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:31.268085 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.268787 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.270419 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.270953 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.271071 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:31.274401 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:31.274406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:31.274409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:31.274438 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:31.276680 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.277331 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.277334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.277336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.277466 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.277785 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.277788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.277790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.278016 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:31.278114 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:31.278708 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 19:39:31.278860 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:39:31.280010 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:31.263914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33167 (local address 127.2.164.67:57681)
0926 19:39:31.264585 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:31.264628 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:31.266249 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:31.266869 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:31.266877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:31.267385 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:31.267932 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:31.267949 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.268923 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.268926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:31.270266 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:31.270277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.271157 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.271164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:31.271363 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:31.272038 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:31.272069 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:31.274229 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:31.276803 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:31.276810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:31.276822 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:31.277242 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:31.277543 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:31.277546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:31.277549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:31.277709 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:31.278100 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:31.278105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:31.278381 (+ 276us) client_negotiation.cc:769] Sending connection context
0926 19:39:31.278614 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 19:39:31.278949 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":246,"threads_started":1}
I20260926 19:39:31.281222 6751 heartbeater.cc:347] Connected to a master server at 127.2.164.126:33167
I20260926 19:39:31.281461 6751 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:31.282092 6751 heartbeater.cc:511] Master 127.2.164.126:33167 requested a full tablet report, sending...
I20260926 19:39:31.283330 6295 ts_manager.cc:194] Registered new tserver with Master: 9c15f859b4c543b187662c4ac69c34dc (127.2.164.67:33143)
I20260926 19:39:31.284343 6295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:57681
I20260926 19:39:31.284876 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:39:31 dist-test-slave-mrw4 krb5kdc[6250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 19:39:31 dist-test-slave-mrw4 krb5kdc[6250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:31.325979 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:31.313010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.2.164.65:39155)
0926 19:39:31.313496 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:31.313504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:31.313559 (+ 55us) server_negotiation.cc:407] Connection header received
0926 19:39:31.313705 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:31.313712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:31.313923 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:31.314075 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:31.315296 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.316255 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.317442 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.317980 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.318075 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:31.321348 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:31.321362 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:31.321407 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:31.321446 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:31.323891 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.324446 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.324453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.324455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.324568 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.324825 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.324828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.324829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.325128 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:31.325385 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:31.325535 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 19:39:31.325709 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":244,"threads_started":1}
Sep 26 19:39:31 dist-test-slave-mrw4 krb5kdc[6250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:31.341200 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:31.329706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.2.164.126:33167)
0926 19:39:31.329898 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:31.329903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:31.330083 (+ 180us) server_negotiation.cc:407] Connection header received
0926 19:39:31.330214 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:31.330218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:31.330286 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:31.330361 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:31.331376 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.332050 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.333217 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.333605 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.333698 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:31.337036 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:31.337040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:31.337043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:31.337072 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:31.339189 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.339734 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.339737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.339739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.339845 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.340096 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.340099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.340101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.340365 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:31.340538 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:31.340749 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 19:39:31.340886 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 19:39:31.341753 6295 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33694
I20260926 19:39:31.342617 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 6356
I20260926 19:39:31.353704 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:31.343401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.2.164.65:39155)
0926 19:39:31.343570 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:31.343575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:31.343730 (+ 155us) server_negotiation.cc:407] Connection header received
0926 19:39:31.343873 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:31.343878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:31.343938 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:31.344010 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:31.345045 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.345806 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.346927 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.347310 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.347430 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:31.349341 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:31.349347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:31.349351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:31.349388 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:31.351513 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.352072 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.352077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.352081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.352189 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.352490 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.352494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.352498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.352718 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:31.352910 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:31.353210 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 19:39:31.353410 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:39:31.354204 6479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:31.432700 6484 heartbeater.cc:503] Master 127.2.164.126:33167 was elected leader, sending a full tablet report...
I20260926 19:39:31.433432 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 6356
W20260926 19:39:31.445891 6280 connection.cc:535] server connection from 127.2.164.65:46839 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 19:39:31.446190 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 6490
Sep 26 19:39:31 dist-test-slave-mrw4 krb5kdc[6250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:39:31.463090 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:31.446994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.2.164.66:40017)
0926 19:39:31.447479 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:31.447488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:31.447514 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:39:31.447571 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:31.447577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:31.447787 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:31.447928 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:31.448985 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.450029 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.451903 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.452691 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.452809 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:31.457929 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:31.457943 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:31.457963 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:31.458003 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:31.460685 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.461290 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.461297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.461299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.461446 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.461764 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.461767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.461769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.462082 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:31.462297 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:31.462570 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 19:39:31.462754 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":236,"threads_started":1}
I20260926 19:39:31.464298 6613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:31.543216 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 6490
I20260926 19:39:31.555395 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 6623
Sep 26 19:39:31 dist-test-slave-mrw4 krb5kdc[6250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:39:31.571339 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:31.556310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.2.164.67:33143)
0926 19:39:31.556913 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:31.556920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:31.556947 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:39:31.557000 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:31.557006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:31.557242 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:31.557408 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:31.558843 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.560191 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.561540 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:31.562088 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:31.562199 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:31.566207 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:31.566218 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:31.566231 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:31.566266 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:31.568783 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.569419 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.569426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.569428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.569549 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:31.569896 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:31.569900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:31.569902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:31.570212 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:31.570492 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:31.570772 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 19:39:31.571015 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":323,"threads_started":1}
I20260926 19:39:31.572598 6746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:31.652716 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 6623
I20260926 19:39:31.664688 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 6263
I20260926 19:39:31.665735 6325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:31.763260 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 6263
2026-09-26T19:39:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3490 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:31 dist-test-slave-mrw4 krb5kdc[6770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:31 dist-test-slave-mrw4 krb5kdc[6770](info): set up 2 sockets
Sep 26 19:39:31 dist-test-slave-mrw4 krb5kdc[6770](info): commencing operation
krb5kdc: starting...
W20260926 19:39:32.764932 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:32 dist-test-slave-mrw4 krb5kdc[6770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:39:32.934471 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36947
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:36947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:33.233057 6783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:33.233322 6783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:33.233448 6783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:33.240274 6783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:33.240365 6783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:33.240435 6783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:33.240509 6783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:33.240568 6783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:33.249708 6783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:36947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:33.250808 6783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:33.251992 6783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:33.262266 6791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:33.262328 6788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:33.262348 6789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:33.262919 6783 server_base.cc:1034] running on GCE node
I20260926 19:39:33.263574 6783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:33.264792 6783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:33.265985 6783 hybrid_clock.cc:648] HybridClock initialized: now 1790451573265931 us; error 44 us; skew 500 ppm
I20260926 19:39:33.266268 6783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:33 dist-test-slave-mrw4 krb5kdc[6770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451573, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:33.272231 6783 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:39:33.273367 6783 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:33.273455 6783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:33.273736 6783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:33.276444 6783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "e0d5018c90b74d258aeeca9d3caf6b67"
format_stamp: "Formatted at 2026-09-26 19:39:33 on dist-test-slave-mrw4"
server_key: "5b3ab1ae9f4d1180111bc37fcac4c1f7"
server_key_iv: "6a39e4331c14e1e941adba2407b75fd0"
server_key_version: "encryptionkey@0"
I20260926 19:39:33.277190 6783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "e0d5018c90b74d258aeeca9d3caf6b67"
format_stamp: "Formatted at 2026-09-26 19:39:33 on dist-test-slave-mrw4"
server_key: "5b3ab1ae9f4d1180111bc37fcac4c1f7"
server_key_iv: "6a39e4331c14e1e941adba2407b75fd0"
server_key_version: "encryptionkey@0"
I20260926 19:39:33.281852 6783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:39:33.285122 6797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:33.286417 6783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:39:33.286578 6783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "e0d5018c90b74d258aeeca9d3caf6b67"
format_stamp: "Formatted at 2026-09-26 19:39:33 on dist-test-slave-mrw4"
server_key: "5b3ab1ae9f4d1180111bc37fcac4c1f7"
server_key_iv: "6a39e4331c14e1e941adba2407b75fd0"
server_key_version: "encryptionkey@0"
I20260926 19:39:33.286751 6783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:33.301429 6783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:33.304855 6783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:33.305045 6783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:33.320763 6783 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:36947
I20260926 19:39:33.320784 6848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:36947 every 8 connection(s)
I20260926 19:39:33.321564 6783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:33.324846 6849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:33.329140 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 6783
I20260926 19:39:33.329380 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:33.330119 2705 external_mini_cluster.cc:1428] Setting key 71109b84b5673baa3b31e955e0eeebdd
I20260926 19:39:33.335575 6849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67: Bootstrap starting.
I20260926 19:39:33.338871 6849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:33.339978 6849 log.cc:824] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:33.342423 6849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67: No bootstrap required, opened a new log
Sep 26 19:39:33 dist-test-slave-mrw4 krb5kdc[6770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:33.349650 6849 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d5018c90b74d258aeeca9d3caf6b67" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36947 } }
I20260926 19:39:33.349960 6849 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:33.350075 6849 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d5018c90b74d258aeeca9d3caf6b67, State: Initialized, Role: FOLLOWER
I20260926 19:39:33.350591 6849 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d5018c90b74d258aeeca9d3caf6b67" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36947 } }
I20260926 19:39:33.350764 6852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:33.331919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.2.164.126:36947)
0926 19:39:33.332638 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:33.332648 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:33.332687 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:39:33.334387 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:33.334407 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:33.334790 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:33.335076 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:33.336729 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:33.338090 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:33.339770 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:33.340493 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:33.340680 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:33.344067 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:33.344074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:33.344091 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:33.344138 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:33.346968 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:33.347594 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:33.347600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:33.347609 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:33.347761 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:33.348105 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:33.348109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:33.348111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:33.348607 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:33.348872 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:33.349309 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 19:39:33.349671 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":319,"threads_started":1}
I20260926 19:39:33.350800 6849 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:33.350919 6849 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:33.351099 6849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:33.353564 6849 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d5018c90b74d258aeeca9d3caf6b67" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36947 } }
I20260926 19:39:33.353945 6849 leader_election.cc:302] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d5018c90b74d258aeeca9d3caf6b67; no voters:
I20260926 19:39:33.354410 6849 leader_election.cc:288] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:33.354519 6854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:33.355075 6854 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [term 1 LEADER]: Becoming Leader. State: Replica: e0d5018c90b74d258aeeca9d3caf6b67, State: Running, Role: LEADER
I20260926 19:39:33.355545 6854 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d5018c90b74d258aeeca9d3caf6b67" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36947 } }
I20260926 19:39:33.356115 6849 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:33.357903 6855 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0d5018c90b74d258aeeca9d3caf6b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d5018c90b74d258aeeca9d3caf6b67" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36947 } } }
I20260926 19:39:33.358340 6855 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:33.358237 6856 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d5018c90b74d258aeeca9d3caf6b67. Latest consensus state: current_term: 1 leader_uuid: "e0d5018c90b74d258aeeca9d3caf6b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d5018c90b74d258aeeca9d3caf6b67" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36947 } } }
I20260926 19:39:33.358506 6856 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:33.359323 6861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:33.364053 6861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:33.372526 6861 catalog_manager.cc:1347] Generated new cluster ID: 1e52c928362c4a26a9250f713a9a6501
I20260926 19:39:33.372628 6861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:33.397521 6861 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:33.398710 6861 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:33.414680 6861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67: Generated new TSK 0
I20260926 19:39:33.415656 6861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:39:33.470710 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36947
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:33.777894 6875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:33.778146 6875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:33.778265 6875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:33.785295 6875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:33.785393 6875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:33.785533 6875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:33.794811 6875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:33.796023 6875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:33.797231 6875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:33.809505 6883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:33.809584 6880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:33.809564 6881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:33.810398 6875 server_base.cc:1034] running on GCE node
I20260926 19:39:33.810918 6875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:33.811749 6875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:33.812950 6875 hybrid_clock.cc:648] HybridClock initialized: now 1790451573812902 us; error 44 us; skew 500 ppm
I20260926 19:39:33.813280 6875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:33 dist-test-slave-mrw4 krb5kdc[6770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451573, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:33.816046 6875 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:39:33.817066 6875 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:33.817188 6875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:33.817488 6875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:33.820135 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "abcbd98d3d774a7797ed0880126a3433"
format_stamp: "Formatted at 2026-09-26 19:39:33 on dist-test-slave-mrw4"
server_key: "e76e14bbb16abd0a9e664d112147fdc8"
server_key_iv: "892c74699534f66d5d45141efbd8e488"
server_key_version: "encryptionkey@0"
I20260926 19:39:33.820856 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "abcbd98d3d774a7797ed0880126a3433"
format_stamp: "Formatted at 2026-09-26 19:39:33 on dist-test-slave-mrw4"
server_key: "e76e14bbb16abd0a9e664d112147fdc8"
server_key_iv: "892c74699534f66d5d45141efbd8e488"
server_key_version: "encryptionkey@0"
I20260926 19:39:33.825412 6875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260926 19:39:33.828424 6889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:33.829643 6875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:39:33.829780 6875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "abcbd98d3d774a7797ed0880126a3433"
format_stamp: "Formatted at 2026-09-26 19:39:33 on dist-test-slave-mrw4"
server_key: "e76e14bbb16abd0a9e664d112147fdc8"
server_key_iv: "892c74699534f66d5d45141efbd8e488"
server_key_version: "encryptionkey@0"
I20260926 19:39:33.829969 6875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:33.852571 6875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:33.856489 6875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:33.856730 6875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:33.857671 6875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:33.858857 6875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:33.858949 6875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:33.859045 6875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:33.859117 6875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:33.887020 6875 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:39059
I20260926 19:39:33.887079 7001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:39059 every 8 connection(s)
I20260926 19:39:33.887799 6875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:33.897832 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 6875
I20260926 19:39:33.898012 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:33.898576 2705 external_mini_cluster.cc:1428] Setting key cd443e919b409720b44c673b0b6dd7e2
Sep 26 19:39:33 dist-test-slave-mrw4 krb5kdc[6770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451573, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:33.905622 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:33.890036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:33599 (local address 127.2.164.126:36947)
0926 19:39:33.890429 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:33.890434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:33.891442 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 19:39:33.893105 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:33.893109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:33.893216 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:33.893306 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:33.894659 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:33.895351 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:33.896752 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:33.897100 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:33.897221 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:33.900406 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:33.900411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:33.900414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:33.900442 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:33.902758 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:33.903508 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:33.903511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:33.903514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:33.903612 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:33.903988 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:33.903990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:33.903992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:33.904226 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:33.904326 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:33.905038 (+ 712us) server_negotiation.cc:299] Negotiation successful
0926 19:39:33.905271 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":197,"threads_started":1}
I20260926 19:39:33.906415 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:33.890614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36947 (local address 127.2.164.65:33599)
0926 19:39:33.891289 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:33.891324 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:33.892871 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:33.893488 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:33.893495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:33.893972 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:33.894511 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:33.894527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:33.895458 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:33.895461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:33.896651 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:33.896662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:33.897284 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:33.897288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:33.897458 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:33.898025 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:33.898044 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:33.900224 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:33.902880 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:33.902888 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:33.902902 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:33.903416 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:33.903695 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:33.903699 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:33.903702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:33.903899 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:33.904327 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:33.904333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:33.904690 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 19:39:33.904960 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 19:39:33.905385 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":276,"threads_started":1}
I20260926 19:39:33.907732 7002 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36947
I20260926 19:39:33.907982 7002 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:33.908545 7002 heartbeater.cc:511] Master 127.2.164.126:36947 requested a full tablet report, sending...
I20260926 19:39:33.910410 6814 ts_manager.cc:194] Registered new tserver with Master: abcbd98d3d774a7797ed0880126a3433 (127.2.164.65:39059)
I20260926 19:39:33.911828 6814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:33599
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:39:33.936721 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36947
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:34.233592 7008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:34.233887 7008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:34.234035 7008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:34.240877 7008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:34.240985 7008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:34.241146 7008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:34.250398 7008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:34.251632 7008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:34.252859 7008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:34.264730 7016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:34.264703 7014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:34.264983 7013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:34.265588 7008 server_base.cc:1034] running on GCE node
I20260926 19:39:34.266090 7008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:34.266889 7008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:34.268076 7008 hybrid_clock.cc:648] HybridClock initialized: now 1790451574268028 us; error 43 us; skew 500 ppm
I20260926 19:39:34.268348 7008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:34 dist-test-slave-mrw4 krb5kdc[6770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:34.271225 7008 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:39:34.272235 7008 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:34.272316 7008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:34.272557 7008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:34.275249 7008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "c8996540ba714394953d73bafc2fb05e"
format_stamp: "Formatted at 2026-09-26 19:39:34 on dist-test-slave-mrw4"
server_key: "99c8f46ab3ce73a596fa79820d1511a4"
server_key_iv: "929c0cc5b260092cb02083133a99a328"
server_key_version: "encryptionkey@0"
I20260926 19:39:34.275954 7008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "c8996540ba714394953d73bafc2fb05e"
format_stamp: "Formatted at 2026-09-26 19:39:34 on dist-test-slave-mrw4"
server_key: "99c8f46ab3ce73a596fa79820d1511a4"
server_key_iv: "929c0cc5b260092cb02083133a99a328"
server_key_version: "encryptionkey@0"
I20260926 19:39:34.280531 7008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:39:34.283762 7022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:34.285012 7008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:39:34.285148 7008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "c8996540ba714394953d73bafc2fb05e"
format_stamp: "Formatted at 2026-09-26 19:39:34 on dist-test-slave-mrw4"
server_key: "99c8f46ab3ce73a596fa79820d1511a4"
server_key_iv: "929c0cc5b260092cb02083133a99a328"
server_key_version: "encryptionkey@0"
I20260926 19:39:34.285394 7008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:34.308254 7008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:34.312602 7008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:34.312870 7008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:34.314357 7008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:34.316042 7008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:34.316157 7008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:34.316272 7008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:34.316344 7008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:34.350756 7008 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:40129
I20260926 19:39:34.350847 7134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:40129 every 8 connection(s)
I20260926 19:39:34.351521 7008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:34.361243 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 7008
I20260926 19:39:34.361423 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:34.361985 2705 external_mini_cluster.cc:1428] Setting key b3e2de4099e4598fbcd053a8273f3b8e
Sep 26 19:39:34 dist-test-slave-mrw4 krb5kdc[6770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:34.369681 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:34.353684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:43813 (local address 127.2.164.126:36947)
0926 19:39:34.353851 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:34.353855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:34.354973 (+ 1118us) server_negotiation.cc:407] Connection header received
0926 19:39:34.356786 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:34.356792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:34.356863 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:34.356938 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:34.358334 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.359018 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.360457 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.360845 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.360929 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:34.364617 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:34.364621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:34.364625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:34.364654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:34.366987 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:34.367629 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:34.367635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:34.367638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:34.367756 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:34.368124 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:34.368136 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:34.368140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:34.368376 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:34.368512 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:34.369135 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 19:39:34.369371 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:39:34.370571 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:34.354150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36947 (local address 127.2.164.66:43813)
0926 19:39:34.354820 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:34.354856 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:34.356516 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:34.357142 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:34.357150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:34.357684 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:34.358208 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:34.358224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.359147 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.359158 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:34.360356 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:34.360367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.361086 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.361093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:34.361391 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:34.362058 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:34.362086 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:34.364410 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:34.367111 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:34.367119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:34.367134 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:34.367524 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:34.367853 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:34.367858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:34.367862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:34.368027 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:34.368491 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:34.368498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:34.368801 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 19:39:34.369025 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 19:39:34.369519 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":273,"threads_started":1}
I20260926 19:39:34.371809 7135 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36947
I20260926 19:39:34.372071 7135 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:34.372646 7135 heartbeater.cc:511] Master 127.2.164.126:36947 requested a full tablet report, sending...
I20260926 19:39:34.373893 6814 ts_manager.cc:194] Registered new tserver with Master: c8996540ba714394953d73bafc2fb05e (127.2.164.66:40129)
I20260926 19:39:34.374662 6814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:43813
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:39:34.400192 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36947
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:34.696144 7140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:34.696393 7140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:34.696514 7140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:34.703382 7140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:34.703478 7140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:34.703625 7140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:34.712843 7140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:34.714098 7140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:34.715276 7140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:34.727104 7145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:34.727078 7146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:34.727164 7148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:34.727717 7140 server_base.cc:1034] running on GCE node
I20260926 19:39:34.728279 7140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:34.729133 7140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:34.730340 7140 hybrid_clock.cc:648] HybridClock initialized: now 1790451574730294 us; error 42 us; skew 500 ppm
I20260926 19:39:34.730621 7140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:34 dist-test-slave-mrw4 krb5kdc[6770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:34.733556 7140 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:39:34.734603 7140 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:34.734691 7140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:34.734972 7140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:34.737715 7140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "c69512bd9a174dfd89bffb5cb43a8273"
format_stamp: "Formatted at 2026-09-26 19:39:34 on dist-test-slave-mrw4"
server_key: "5aa5fc34f803c20a0ca2432ae1c1ba36"
server_key_iv: "13984de295d2e6f1d17d6fef70ee5135"
server_key_version: "encryptionkey@0"
I20260926 19:39:34.738466 7140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "c69512bd9a174dfd89bffb5cb43a8273"
format_stamp: "Formatted at 2026-09-26 19:39:34 on dist-test-slave-mrw4"
server_key: "5aa5fc34f803c20a0ca2432ae1c1ba36"
server_key_iv: "13984de295d2e6f1d17d6fef70ee5135"
server_key_version: "encryptionkey@0"
I20260926 19:39:34.743065 7140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:39:34.746160 7154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:34.747287 7140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:39:34.747431 7140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "c69512bd9a174dfd89bffb5cb43a8273"
format_stamp: "Formatted at 2026-09-26 19:39:34 on dist-test-slave-mrw4"
server_key: "5aa5fc34f803c20a0ca2432ae1c1ba36"
server_key_iv: "13984de295d2e6f1d17d6fef70ee5135"
server_key_version: "encryptionkey@0"
I20260926 19:39:34.747627 7140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:34.767280 7140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:34.771088 7140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:34.771355 7140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:34.772224 7140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:34.773451 7140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:34.773535 7140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:34.773629 7140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:34.773701 7140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:34.801196 7140 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:44127
I20260926 19:39:34.801227 7266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:44127 every 8 connection(s)
I20260926 19:39:34.801923 7140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:34.804528 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 7140
I20260926 19:39:34.804689 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:34.805225 2705 external_mini_cluster.cc:1428] Setting key 708fd61ed229e82026886900cbeb901c
Sep 26 19:39:34 dist-test-slave-mrw4 krb5kdc[6770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451574, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:34.820132 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:34.804023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:45323 (local address 127.2.164.126:36947)
0926 19:39:34.804193 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:34.804198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:34.805445 (+ 1247us) server_negotiation.cc:407] Connection header received
0926 19:39:34.807817 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:34.807822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:34.807900 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:34.807980 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:34.809652 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.810380 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.811824 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.812215 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.812315 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:34.815328 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:34.815333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:34.815337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:34.815366 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:34.817494 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:34.818122 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:34.818126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:34.818129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:34.818231 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:34.818598 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:34.818601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:34.818603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:34.818855 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:34.818978 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:34.819612 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 19:39:34.819808 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:39:34.820987 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:34.804476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36947 (local address 127.2.164.67:45323)
0926 19:39:34.805277 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:34.805324 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:34.807584 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:34.808224 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:34.808235 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:34.808786 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:34.809508 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:34.809527 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.810519 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.810525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:34.811695 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:34.811706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.812406 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.812413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:34.812620 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:34.813213 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:34.813238 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:34.815141 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:34.817606 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:34.817615 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:34.817630 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:34.818007 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:34.818323 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:34.818328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:34.818332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:34.818484 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:34.818973 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:34.818980 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:34.819261 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 19:39:34.819467 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 19:39:34.819909 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":299,"threads_started":1}
I20260926 19:39:34.822177 7267 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36947
I20260926 19:39:34.822425 7267 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:34.822979 7267 heartbeater.cc:511] Master 127.2.164.126:36947 requested a full tablet report, sending...
I20260926 19:39:34.824159 6814 ts_manager.cc:194] Registered new tserver with Master: c69512bd9a174dfd89bffb5cb43a8273 (127.2.164.67:44127)
I20260926 19:39:34.824874 6814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:45323
I20260926 19:39:34.833118 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:39:34.847959 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:34.837212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.2.164.126:36947)
0926 19:39:34.837380 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:34.837387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:34.837816 (+ 429us) server_negotiation.cc:407] Connection header received
0926 19:39:34.837978 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:34.837982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:34.838058 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:34.838136 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:34.839515 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.840191 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.841436 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.841865 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.841955 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:34.843903 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:34.843907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:34.843910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:34.843937 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:34.846040 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:34.846543 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:34.846546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:34.846548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:34.846655 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:34.846916 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:34.846919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:34.846921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:34.847181 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:34.847340 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:34.847569 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 19:39:34.847705 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:39:34.851591 6814 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:39:34.858099 6814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:34.889493 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:34.868914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:39059 (local address 127.0.0.1:48484)
0926 19:39:34.880872 (+ 11958us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:34.880890 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:34.881003 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:34.882002 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:34.882010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:34.882058 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:34.882518 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:34.882534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.883893 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.883899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:34.885809 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:34.885819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.887598 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.887604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:34.888674 (+ 1070us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:34.888730 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 19:39:34.888851 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 19:39:34.889028 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11855,"mutex_wait_us":19,"spinlock_wait_cycles":896,"thread_start_us":10497,"threads_started":1}
I20260926 19:39:34.890223 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:34.868787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:40129 (local address 127.0.0.1:36332)
0926 19:39:34.879810 (+ 11023us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:34.879860 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:34.880037 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:34.882002 (+ 1965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:34.882009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:34.882054 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:34.882953 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:34.882965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.884749 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.884755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:34.887169 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:34.887181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.888914 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.888920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:34.889068 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:34.889103 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:39:34.889202 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 19:39:34.889317 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10763,"mutex_wait_us":141,"spinlock_wait_cycles":1792,"thread_start_us":12623,"threads_started":1}
I20260926 19:39:34.890280 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:34.869837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.2.164.67:44127)
0926 19:39:34.880512 (+ 10675us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:34.880522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:34.881617 (+ 1095us) server_negotiation.cc:407] Connection header received
0926 19:39:34.881881 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:34.881889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:34.882120 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:34.882285 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:34.883003 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.884588 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.887245 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.888631 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.888728 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:34.888937 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:34.889832 (+ 895us) server_negotiation.cc:299] Negotiation successful
0926 19:39:34.889980 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10510,"thread_start_us":10373,"threads_started":1}
I20260926 19:39:34.890656 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:34.869370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.2.164.65:39059)
0926 19:39:34.881464 (+ 12094us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:34.881472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:34.881495 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:39:34.881586 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:34.881593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:34.881793 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:34.882434 (+ 641us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:34.882685 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.883777 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.885945 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.887155 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.887281 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:34.887481 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:34.890015 (+ 2534us) server_negotiation.cc:299] Negotiation successful
0926 19:39:34.890178 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11955,"thread_start_us":11873,"threads_started":1}
I20260926 19:39:34.890712 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:34.868924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.2.164.66:40129)
0926 19:39:34.881154 (+ 12230us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:34.881189 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:34.881219 (+ 30us) server_negotiation.cc:407] Connection header received
0926 19:39:34.881321 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:34.881328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:34.881590 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:34.881774 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:34.883105 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.884588 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.887322 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.888669 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.889453 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:34.889623 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:34.889698 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 19:39:34.889837 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12080,"thread_start_us":11992,"threads_started":1}
I20260926 19:39:34.891232 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:34.869247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:44127 (local address 127.0.0.1:37578)
0926 19:39:34.881504 (+ 12257us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:34.881520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:34.881704 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:34.882447 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:34.882451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:34.882479 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:34.882810 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:34.882818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.884751 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.884755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:34.887061 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:34.887075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:34.889201 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:34.889207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:34.889469 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:34.889503 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:39:34.890649 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0926 19:39:34.890879 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12127,"mutex_wait_us":120,"thread_start_us":11412,"threads_started":1}
I20260926 19:39:34.894248 7070 tablet_service.cc:1467] Processing CreateTablet for tablet c7b6bd11d3d34310b430bcae9d37e8f5 (DEFAULT_TABLE table=test-table [id=3b5d1717972646c8a8912cc0483b98a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:34.894708 6937 tablet_service.cc:1467] Processing CreateTablet for tablet c7b6bd11d3d34310b430bcae9d37e8f5 (DEFAULT_TABLE table=test-table [id=3b5d1717972646c8a8912cc0483b98a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:34.894707 7202 tablet_service.cc:1467] Processing CreateTablet for tablet c7b6bd11d3d34310b430bcae9d37e8f5 (DEFAULT_TABLE table=test-table [id=3b5d1717972646c8a8912cc0483b98a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:34.895370 7070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b6bd11d3d34310b430bcae9d37e8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:34.895939 6937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b6bd11d3d34310b430bcae9d37e8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:34.895941 7202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b6bd11d3d34310b430bcae9d37e8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:34.905738 7285 tablet_bootstrap.cc:492] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: Bootstrap starting.
I20260926 19:39:34.908768 7285 tablet_bootstrap.cc:654] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:34.909370 7287 tablet_bootstrap.cc:492] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: Bootstrap starting.
I20260926 19:39:34.910092 7285 log.cc:824] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:34.912514 7287 tablet_bootstrap.cc:654] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:34.912945 7285 tablet_bootstrap.cc:492] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: No bootstrap required, opened a new log
I20260926 19:39:34.913216 7285 ts_tablet_manager.cc:1397] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260926 19:39:34.913304 7286 tablet_bootstrap.cc:492] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: Bootstrap starting.
I20260926 19:39:34.913662 7287 log.cc:824] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:34.914413 7002 heartbeater.cc:503] Master 127.2.164.126:36947 was elected leader, sending a full tablet report...
I20260926 19:39:34.916190 7286 tablet_bootstrap.cc:654] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:34.917294 7286 log.cc:824] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:34.917858 7287 tablet_bootstrap.cc:492] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: No bootstrap required, opened a new log
I20260926 19:39:34.918411 7287 ts_tablet_manager.cc:1397] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 19:39:34.922681 7285 raft_consensus.cc:348] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8996540ba714394953d73bafc2fb05e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40129 } } peers { permanent_uuid: "abcbd98d3d774a7797ed0880126a3433" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39059 } } peers { permanent_uuid: "c69512bd9a174dfd89bffb5cb43a8273" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44127 } }
I20260926 19:39:34.923105 7285 raft_consensus.cc:374] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:34.923168 7286 tablet_bootstrap.cc:492] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: No bootstrap required, opened a new log
I20260926 19:39:34.923202 7285 raft_consensus.cc:729] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8996540ba714394953d73bafc2fb05e, State: Initialized, Role: FOLLOWER
I20260926 19:39:34.923398 7286 ts_tablet_manager.cc:1397] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260926 19:39:34.923844 7285 consensus_queue.cc:260] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8996540ba714394953d73bafc2fb05e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40129 } } peers { permanent_uuid: "abcbd98d3d774a7797ed0880126a3433" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39059 } } peers { permanent_uuid: "c69512bd9a174dfd89bffb5cb43a8273" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44127 } }
I20260926 19:39:34.929224 7287 raft_consensus.cc:348] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8996540ba714394953d73bafc2fb05e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40129 } } peers { permanent_uuid: "abcbd98d3d774a7797ed0880126a3433" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39059 } } peers { permanent_uuid: "c69512bd9a174dfd89bffb5cb43a8273" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44127 } }
I20260926 19:39:34.929620 7287 raft_consensus.cc:374] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:34.929713 7287 raft_consensus.cc:729] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abcbd98d3d774a7797ed0880126a3433, State: Initialized, Role: FOLLOWER
I20260926 19:39:34.929948 7135 heartbeater.cc:503] Master 127.2.164.126:36947 was elected leader, sending a full tablet report...
I20260926 19:39:34.930456 7285 ts_tablet_manager.cc:1428] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: Time spent starting tablet: real 0.017s user 0.002s sys 0.014s
I20260926 19:39:34.930238 7286 raft_consensus.cc:348] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8996540ba714394953d73bafc2fb05e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40129 } } peers { permanent_uuid: "abcbd98d3d774a7797ed0880126a3433" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39059 } } peers { permanent_uuid: "c69512bd9a174dfd89bffb5cb43a8273" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44127 } }
I20260926 19:39:34.930372 7287 consensus_queue.cc:260] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8996540ba714394953d73bafc2fb05e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40129 } } peers { permanent_uuid: "abcbd98d3d774a7797ed0880126a3433" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39059 } } peers { permanent_uuid: "c69512bd9a174dfd89bffb5cb43a8273" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44127 } }
I20260926 19:39:34.930583 7286 raft_consensus.cc:374] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:34.930688 7286 raft_consensus.cc:729] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c69512bd9a174dfd89bffb5cb43a8273, State: Initialized, Role: FOLLOWER
I20260926 19:39:34.931208 7286 consensus_queue.cc:260] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8996540ba714394953d73bafc2fb05e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40129 } } peers { permanent_uuid: "abcbd98d3d774a7797ed0880126a3433" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39059 } } peers { permanent_uuid: "c69512bd9a174dfd89bffb5cb43a8273" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44127 } }
I20260926 19:39:34.932068 7287 ts_tablet_manager.cc:1428] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260926 19:39:34.932981 7267 heartbeater.cc:503] Master 127.2.164.126:36947 was elected leader, sending a full tablet report...
I20260926 19:39:34.933578 7286 ts_tablet_manager.cc:1428] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
W20260926 19:39:35.053221 7268 tablet.cc:2369] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:39:35.103161 7136 tablet.cc:2369] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:39:35.139766 7003 tablet.cc:2369] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:35.154269 7293 raft_consensus.cc:482] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:35.154548 7293 raft_consensus.cc:504] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8996540ba714394953d73bafc2fb05e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40129 } } peers { permanent_uuid: "abcbd98d3d774a7797ed0880126a3433" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39059 } } peers { permanent_uuid: "c69512bd9a174dfd89bffb5cb43a8273" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44127 } }
I20260926 19:39:35.156289 7293 leader_election.cc:288] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8996540ba714394953d73bafc2fb05e (127.2.164.66:40129), abcbd98d3d774a7797ed0880126a3433 (127.2.164.65:39059)
I20260926 19:39:35.162298 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:35.156725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:42925 (local address 127.2.164.66:40129)
0926 19:39:35.156912 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:35.156917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:35.156937 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:39:35.157129 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:35.157134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:35.157240 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:35.157331 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:35.157727 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.158628 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.160389 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.161522 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.161637 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:35.161689 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:35.162020 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 19:39:35.162119 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:39:35.162393 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:35.156488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:40129 (local address 127.2.164.67:42925)
0926 19:39:35.156810 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:35.156833 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:35.156955 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:35.157367 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:35.157372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:35.157408 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:35.157582 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:35.157594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.158749 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.158754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:35.160232 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:35.160243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.161657 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.161664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:35.161870 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:35.161903 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:39:35.161999 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 19:39:35.162127 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":2}
I20260926 19:39:35.163257 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:35.156864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:51427 (local address 127.2.164.65:39059)
0926 19:39:35.157062 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:35.157069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:35.157551 (+ 482us) server_negotiation.cc:407] Connection header received
0926 19:39:35.157685 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:35.157690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:35.157756 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:35.157822 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:35.158407 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.159324 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.161348 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.162402 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.162542 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:35.162607 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:35.162898 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 19:39:35.163030 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:39:35.163285 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:35.156714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:39059 (local address 127.2.164.67:51427)
0926 19:39:35.157453 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:35.157469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:35.157566 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:35.157855 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:35.157859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:35.157882 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:35.158237 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:35.158247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.159488 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.159493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:35.161183 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:35.161198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.162663 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.162667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:35.162773 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:35.162796 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:39:35.162875 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 19:39:35.162990 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":408,"threads_started":1}
I20260926 19:39:35.163331 7090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b6bd11d3d34310b430bcae9d37e8f5" candidate_uuid: "c69512bd9a174dfd89bffb5cb43a8273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8996540ba714394953d73bafc2fb05e" is_pre_election: true
I20260926 19:39:35.163847 7090 raft_consensus.cc:2454] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c69512bd9a174dfd89bffb5cb43a8273 in term 0.
I20260926 19:39:35.164191 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b6bd11d3d34310b430bcae9d37e8f5" candidate_uuid: "c69512bd9a174dfd89bffb5cb43a8273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abcbd98d3d774a7797ed0880126a3433" is_pre_election: true
I20260926 19:39:35.164397 7157 leader_election.cc:302] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c69512bd9a174dfd89bffb5cb43a8273, c8996540ba714394953d73bafc2fb05e; no voters:
I20260926 19:39:35.164605 6957 raft_consensus.cc:2454] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c69512bd9a174dfd89bffb5cb43a8273 in term 0.
I20260926 19:39:35.164750 7293 raft_consensus.cc:2781] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:35.164882 7293 raft_consensus.cc:482] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:35.164979 7293 raft_consensus.cc:3037] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:35.167085 7293 raft_consensus.cc:504] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8996540ba714394953d73bafc2fb05e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40129 } } peers { permanent_uuid: "abcbd98d3d774a7797ed0880126a3433" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39059 } } peers { permanent_uuid: "c69512bd9a174dfd89bffb5cb43a8273" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44127 } }
I20260926 19:39:35.167637 7293 leader_election.cc:288] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [CANDIDATE]: Term 1 election: Requested vote from peers c8996540ba714394953d73bafc2fb05e (127.2.164.66:40129), abcbd98d3d774a7797ed0880126a3433 (127.2.164.65:39059)
I20260926 19:39:35.168044 7090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b6bd11d3d34310b430bcae9d37e8f5" candidate_uuid: "c69512bd9a174dfd89bffb5cb43a8273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8996540ba714394953d73bafc2fb05e"
I20260926 19:39:35.168081 6957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b6bd11d3d34310b430bcae9d37e8f5" candidate_uuid: "c69512bd9a174dfd89bffb5cb43a8273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abcbd98d3d774a7797ed0880126a3433"
I20260926 19:39:35.168244 6957 raft_consensus.cc:3037] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:35.168246 7090 raft_consensus.cc:3037] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:35.171219 6957 raft_consensus.cc:2454] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c69512bd9a174dfd89bffb5cb43a8273 in term 1.
I20260926 19:39:35.171219 7090 raft_consensus.cc:2454] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c69512bd9a174dfd89bffb5cb43a8273 in term 1.
I20260926 19:39:35.171569 7157 leader_election.cc:302] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c69512bd9a174dfd89bffb5cb43a8273, c8996540ba714394953d73bafc2fb05e; no voters:
I20260926 19:39:35.171835 7293 raft_consensus.cc:2781] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:35.172292 7293 raft_consensus.cc:686] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 1 LEADER]: Becoming Leader. State: Replica: c69512bd9a174dfd89bffb5cb43a8273, State: Running, Role: LEADER
I20260926 19:39:35.172729 7293 consensus_queue.cc:237] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8996540ba714394953d73bafc2fb05e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40129 } } peers { permanent_uuid: "abcbd98d3d774a7797ed0880126a3433" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39059 } } peers { permanent_uuid: "c69512bd9a174dfd89bffb5cb43a8273" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44127 } }
I20260926 19:39:35.177008 6812 catalog_manager.cc:5495] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 reported cstate change: term changed from 0 to 1, leader changed from <none> to c69512bd9a174dfd89bffb5cb43a8273 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "c69512bd9a174dfd89bffb5cb43a8273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8996540ba714394953d73bafc2fb05e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abcbd98d3d774a7797ed0880126a3433" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c69512bd9a174dfd89bffb5cb43a8273" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44127 } health_report { overall_health: HEALTHY } } }
I20260926 19:39:35.259631 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:35.254438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.2.164.67:44127)
0926 19:39:35.254617 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:35.254623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:35.254730 (+ 107us) server_negotiation.cc:407] Connection header received
0926 19:39:35.254941 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:35.254945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:35.255026 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:35.255149 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:35.255495 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.256232 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.257641 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.258035 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.258141 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:35.258576 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:35.259088 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:35.259215 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:35.259322 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 19:39:35.259399 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:39:35.267354 7090 raft_consensus.cc:1261] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e [term 1 FOLLOWER]: Refusing update from remote peer c69512bd9a174dfd89bffb5cb43a8273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:35.267354 6957 raft_consensus.cc:1261] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433 [term 1 FOLLOWER]: Refusing update from remote peer c69512bd9a174dfd89bffb5cb43a8273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:35.268399 7295 consensus_queue.cc:1035] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abcbd98d3d774a7797ed0880126a3433" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:35.268721 7293 consensus_queue.cc:1035] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8996540ba714394953d73bafc2fb05e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:35.279574 7298 mvcc.cc:205] Tried to move back new op lower bound from 7333689652284829696 to 7333689651910963200. Current Snapshot: MvccSnapshot[applied={T|T < 7333689652284829696}]
I20260926 19:39:35.292361 6813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:35.292604 6813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:35.295974 6813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0d5018c90b74d258aeeca9d3caf6b67: Sending DeleteTablet for 3 replicas of tablet c7b6bd11d3d34310b430bcae9d37e8f5
I20260926 19:39:35.297041 7070 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b6bd11d3d34310b430bcae9d37e8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:35 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:36332
I20260926 19:39:35.297264 7202 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b6bd11d3d34310b430bcae9d37e8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:35 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:37578
I20260926 19:39:35.297261 6937 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b6bd11d3d34310b430bcae9d37e8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:35 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:48484
I20260926 19:39:35.299252 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 6875
I20260926 19:39:35.307174 7308 tablet_replica.cc:330] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: stopping tablet replica
I20260926 19:39:35.307610 7308 raft_consensus.cc:2229] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:39:35.308267 7308 raft_consensus.cc:2258] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:35.314361 7309 tablet_replica.cc:330] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: stopping tablet replica
I20260926 19:39:35.314936 7309 raft_consensus.cc:2229] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:35.315433 7309 raft_consensus.cc:2258] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:35.317984 7308 ts_tablet_manager.cc:1905] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:35.320098 7309 ts_tablet_manager.cc:1905] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:35.322886 7307 tablet_replica.cc:330] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: stopping tablet replica
I20260926 19:39:35.323341 7307 raft_consensus.cc:2229] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:35.323872 7307 raft_consensus.cc:2258] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:35.327077 7308 ts_tablet_manager.cc:1918] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:35.327549 7308 log.cc:1196] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/c7b6bd11d3d34310b430bcae9d37e8f5
I20260926 19:39:35.328377 7308 ts_tablet_manager.cc:1939] T c7b6bd11d3d34310b430bcae9d37e8f5 P c69512bd9a174dfd89bffb5cb43a8273: Deleting consensus metadata
I20260926 19:39:35.329595 7307 ts_tablet_manager.cc:1905] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:35.330216 6801 catalog_manager.cc:4841] TS c69512bd9a174dfd89bffb5cb43a8273 (127.2.164.67:44127): tablet c7b6bd11d3d34310b430bcae9d37e8f5 (table test-table [id=3b5d1717972646c8a8912cc0483b98a4]) successfully deleted
Sep 26 19:39:35 dist-test-slave-mrw4 krb5kdc[6770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:35.331899 7309 ts_tablet_manager.cc:1918] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:35.332114 7309 log.cc:1196] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/c7b6bd11d3d34310b430bcae9d37e8f5
I20260926 19:39:35.332593 7309 ts_tablet_manager.cc:1939] T c7b6bd11d3d34310b430bcae9d37e8f5 P abcbd98d3d774a7797ed0880126a3433: Deleting consensus metadata
I20260926 19:39:35.334438 6798 catalog_manager.cc:4841] TS abcbd98d3d774a7797ed0880126a3433 (127.2.164.65:39059): tablet c7b6bd11d3d34310b430bcae9d37e8f5 (table test-table [id=3b5d1717972646c8a8912cc0483b98a4]) successfully deleted
I20260926 19:39:35.337782 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:35.300694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.2.164.65:39059)
0926 19:39:35.300868 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:35.300875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:35.313640 (+ 12765us) server_negotiation.cc:407] Connection header received
0926 19:39:35.314088 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:35.314094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:35.314186 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:35.314262 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:35.316125 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.317144 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.320801 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.321442 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.321537 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:35.331396 (+ 9859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:35.331411 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:35.331427 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:35.331470 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:35.335156 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:35.335767 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:35.335774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:35.335777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:35.335896 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:35.336251 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:35.336260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:35.336263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:35.336594 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:35.336841 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:35.337124 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 19:39:35.337353 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:39:35.338353 6997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:35.343593 7307 ts_tablet_manager.cc:1918] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:35.343751 7307 log.cc:1196] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/c7b6bd11d3d34310b430bcae9d37e8f5
I20260926 19:39:35.344249 7307 ts_tablet_manager.cc:1939] T c7b6bd11d3d34310b430bcae9d37e8f5 P c8996540ba714394953d73bafc2fb05e: Deleting consensus metadata
I20260926 19:39:35.345980 6800 catalog_manager.cc:4841] TS c8996540ba714394953d73bafc2fb05e (127.2.164.66:40129): tablet c7b6bd11d3d34310b430bcae9d37e8f5 (table test-table [id=3b5d1717972646c8a8912cc0483b98a4]) successfully deleted
I20260926 19:39:35.439283 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 6875
I20260926 19:39:35.455996 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 7008
Sep 26 19:39:35 dist-test-slave-mrw4 krb5kdc[6770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:39:35.477110 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:35.457363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.2.164.66:40129)
0926 19:39:35.458455 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:35.458461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:35.458482 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:39:35.458545 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:35.458551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:35.458797 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:35.458904 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:35.460197 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.461284 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.463532 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.464311 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.464429 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:35.469807 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:35.469825 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:35.469847 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:35.469891 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:35.473443 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:35.474242 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:35.474251 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:35.474255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:35.474403 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:35.474796 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:35.474801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:35.474804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:35.475239 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:35.475515 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:35.476544 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0926 19:39:35.476746 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958}
I20260926 19:39:35.477973 7130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:35.575505 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 7008
I20260926 19:39:35.591246 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 7140
Sep 26 19:39:35 dist-test-slave-mrw4 krb5kdc[6770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:39:35.606998 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:35.592101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.2.164.67:44127)
0926 19:39:35.592308 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:35.592315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:35.592337 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:39:35.592431 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:35.592435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:35.592542 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:35.592660 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:35.593902 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.595044 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.596269 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:35.596926 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:35.597062 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:35.601348 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:35.601362 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:35.601382 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:35.601422 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:35.604235 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:35.605032 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:35.605039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:35.605041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:35.605232 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:35.605606 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:35.605609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:35.605611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:35.605958 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:35.606185 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:35.606516 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 19:39:35.606679 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 19:39:35.607764 7262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:35.725864 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 7140
I20260926 19:39:35.740903 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 6783
I20260926 19:39:35.741981 6844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:35.847702 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 6783
2026-09-26T19:39:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4083 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T19:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:35Z Disabled control of system clock
I20260926 19:39:35.903465 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:39687
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:44345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 19:39:36.201407 7318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:36.201678 7318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:36.201788 7318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:36.208889 7318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:36.208988 7318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:36.209033 7318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:36.209108 7318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:36.209177 7318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:36.218102 7318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:39687
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:36.219365 7318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:36.221100 7318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:36.231720 7324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:36.231760 7326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:36.231781 7323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:36.232872 7318 server_base.cc:1034] running on GCE node
I20260926 19:39:36.233609 7318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:36.234858 7318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:36.236044 7318 hybrid_clock.cc:648] HybridClock initialized: now 1790451576236000 us; error 40 us; skew 500 ppm
I20260926 19:39:36.236366 7318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:39:36.238413 7318 webserver.cc:460] Webserver started at http://127.2.164.126:34723/ using document root <none> and password file <none>
I20260926 19:39:36.239022 7318 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:36.239102 7318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:36.239387 7318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:36.242059 7318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "1eb19e72970146c7a86bce84ae802b52"
format_stamp: "Formatted at 2026-09-26 19:39:36 on dist-test-slave-mrw4"
server_key: "5e632aa72dba986253ea9f86f40ed60c"
server_key_iv: "36d4c8675af4ebec827bb4c3a32aaa06"
server_key_version: "encryptionkey@0"
I20260926 19:39:36.242784 7318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "1eb19e72970146c7a86bce84ae802b52"
format_stamp: "Formatted at 2026-09-26 19:39:36 on dist-test-slave-mrw4"
server_key: "5e632aa72dba986253ea9f86f40ed60c"
server_key_iv: "36d4c8675af4ebec827bb4c3a32aaa06"
server_key_version: "encryptionkey@0"
I20260926 19:39:36.247445 7318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:39:36.250733 7332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:36.252039 7318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:39:36.252183 7318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "1eb19e72970146c7a86bce84ae802b52"
format_stamp: "Formatted at 2026-09-26 19:39:36 on dist-test-slave-mrw4"
server_key: "5e632aa72dba986253ea9f86f40ed60c"
server_key_iv: "36d4c8675af4ebec827bb4c3a32aaa06"
server_key_version: "encryptionkey@0"
I20260926 19:39:36.252381 7318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:36.257880 7318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:36.258569 7318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:36.258833 7318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:36.274933 7318 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:39687
I20260926 19:39:36.274962 7383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:39687 every 8 connection(s)
I20260926 19:39:36.276103 7318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:36.277282 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 7318
I20260926 19:39:36.277458 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:36.278230 2705 external_mini_cluster.cc:1428] Setting key 7449008d0790b24879c0b5acde24fc26
I20260926 19:39:36.280375 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:36.285822 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:36.280136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.2.164.126:39687)
0926 19:39:36.281036 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:36.281047 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:36.281084 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:39:36.282587 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:36.282606 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:36.283008 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:36.283324 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:36.284307 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:36.284313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:36.284324 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:36.284338 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:36.284405 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:36.284423 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:36.284578 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:36.284673 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 19:39:36.284904 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":314,"threads_started":1}
I20260926 19:39:36.291024 7384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52: Bootstrap starting.
I20260926 19:39:36.293812 7384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:36.294694 7384 log.cc:824] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:36.297248 7384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52: No bootstrap required, opened a new log
I20260926 19:39:36.303730 7384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb19e72970146c7a86bce84ae802b52" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39687 } }
I20260926 19:39:36.304029 7384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:36.304108 7384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb19e72970146c7a86bce84ae802b52, State: Initialized, Role: FOLLOWER
I20260926 19:39:36.304580 7384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb19e72970146c7a86bce84ae802b52" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39687 } }
I20260926 19:39:36.304774 7384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:36.304884 7384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:36.305013 7384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:36.306937 7384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb19e72970146c7a86bce84ae802b52" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39687 } }
I20260926 19:39:36.307355 7384 leader_election.cc:302] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eb19e72970146c7a86bce84ae802b52; no voters:
I20260926 19:39:36.307843 7384 leader_election.cc:288] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:36.307945 7389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:36.308557 7389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [term 1 LEADER]: Becoming Leader. State: Replica: 1eb19e72970146c7a86bce84ae802b52, State: Running, Role: LEADER
I20260926 19:39:36.309073 7389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb19e72970146c7a86bce84ae802b52" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39687 } }
I20260926 19:39:36.309639 7384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:36.311977 7390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eb19e72970146c7a86bce84ae802b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb19e72970146c7a86bce84ae802b52" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39687 } } }
I20260926 19:39:36.312354 7390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:36.311976 7391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eb19e72970146c7a86bce84ae802b52. Latest consensus state: current_term: 1 leader_uuid: "1eb19e72970146c7a86bce84ae802b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb19e72970146c7a86bce84ae802b52" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39687 } } }
I20260926 19:39:36.312703 7391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:36.318187 7397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:36.322696 7397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:36.330505 7397 catalog_manager.cc:1347] Generated new cluster ID: 67faa7821c3d4802a93709a64b92a586
I20260926 19:39:36.330622 7397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:36.351716 7397 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:36.352144 7397 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:36.359975 7397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52: Generated new TSK 0
I20260926 19:39:36.360739 7397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:39:36.366813 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39687
--builtin_ntp_servers=127.2.164.84:44345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 19:39:36.669150 7408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:36.669435 7408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:36.669582 7408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:36.676599 7408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:36.676692 7408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:36.676862 7408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:36.685571 7408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:36.686672 7408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:36.688271 7408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:36.702893 7413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:36.702893 7414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:36.705842 7416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:36.706588 7408 server_base.cc:1034] running on GCE node
I20260926 19:39:36.707159 7408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:36.708106 7408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:36.709300 7408 hybrid_clock.cc:648] HybridClock initialized: now 1790451576709262 us; error 48 us; skew 500 ppm
I20260926 19:39:36.709599 7408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:39:36.711953 7408 webserver.cc:460] Webserver started at http://127.2.164.65:41317/ using document root <none> and password file <none>
I20260926 19:39:36.712582 7408 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:36.712667 7408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:36.712950 7408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:36.715629 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "827cd5fd73564465af06399391cbf36a"
format_stamp: "Formatted at 2026-09-26 19:39:36 on dist-test-slave-mrw4"
server_key: "add8e62511096146ce05f6cec7ab5397"
server_key_iv: "daab0ffda0ce2f7c62086b24de7e57b8"
server_key_version: "encryptionkey@0"
I20260926 19:39:36.716348 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "827cd5fd73564465af06399391cbf36a"
format_stamp: "Formatted at 2026-09-26 19:39:36 on dist-test-slave-mrw4"
server_key: "add8e62511096146ce05f6cec7ab5397"
server_key_iv: "daab0ffda0ce2f7c62086b24de7e57b8"
server_key_version: "encryptionkey@0"
I20260926 19:39:36.721241 7408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 19:39:36.724390 7422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:36.725714 7408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:39:36.725874 7408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "827cd5fd73564465af06399391cbf36a"
format_stamp: "Formatted at 2026-09-26 19:39:36 on dist-test-slave-mrw4"
server_key: "add8e62511096146ce05f6cec7ab5397"
server_key_iv: "daab0ffda0ce2f7c62086b24de7e57b8"
server_key_version: "encryptionkey@0"
I20260926 19:39:36.726063 7408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:36.731155 7408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:36.731843 7408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:36.732064 7408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:36.732925 7408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:36.734210 7408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:36.734302 7408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:36.734398 7408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:36.734463 7408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:36.766711 7408 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:42959
I20260926 19:39:36.766752 7534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:42959 every 8 connection(s)
I20260926 19:39:36.768034 7408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:36.772996 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 7408
I20260926 19:39:36.773186 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:36.773957 2705 external_mini_cluster.cc:1428] Setting key 87f2cc0f3b234b6ce42fdce4ed8179bd
I20260926 19:39:36.776829 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:36.770603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:34459 (local address 127.2.164.126:39687)
0926 19:39:36.770798 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:36.770804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:36.772157 (+ 1353us) server_negotiation.cc:407] Connection header received
0926 19:39:36.774284 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:36.774291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:36.774372 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:36.774486 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:36.776082 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:36.776088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:36.776093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:36.776112 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:36.776149 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:36.776165 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:36.776287 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:36.776499 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 19:39:36.776555 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:39:36.776993 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39687
--builtin_ntp_servers=127.2.164.84:44345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 19:39:36.777907 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:36.770998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39687 (local address 127.2.164.65:34459)
0926 19:39:36.771998 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:36.772040 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:36.773978 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:36.774886 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:36.774896 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:36.775013 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:36.775757 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:39:36.775761 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:36.775811 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:39:36.775916 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:36.776285 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:36.776296 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:36.776303 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 19:39:36.776559 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 19:39:36.776797 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":291,"threads_started":1}
I20260926 19:39:36.779362 7535 heartbeater.cc:347] Connected to a master server at 127.2.164.126:39687
I20260926 19:39:36.779660 7535 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:36.780325 7535 heartbeater.cc:511] Master 127.2.164.126:39687 requested a full tablet report, sending...
I20260926 19:39:36.782584 7349 ts_manager.cc:194] Registered new tserver with Master: 827cd5fd73564465af06399391cbf36a (127.2.164.65:42959)
W20260926 19:39:37.109611 7538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:37.109879 7538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:37.110006 7538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:37.116945 7538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:37.117046 7538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:37.117223 7538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:37.126012 7538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:37.127262 7538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:37.128995 7538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:37.141486 7543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:37.141603 7544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:37.141886 7546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:37.142237 7538 server_base.cc:1034] running on GCE node
I20260926 19:39:37.142779 7538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:37.143643 7538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:37.144850 7538 hybrid_clock.cc:648] HybridClock initialized: now 1790451577144792 us; error 61 us; skew 500 ppm
I20260926 19:39:37.145152 7538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:39:37.147200 7538 webserver.cc:460] Webserver started at http://127.2.164.66:33131/ using document root <none> and password file <none>
I20260926 19:39:37.147837 7538 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:37.147929 7538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:37.148222 7538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:37.150924 7538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "50651ca54d1e4c81968582f079a7c380"
format_stamp: "Formatted at 2026-09-26 19:39:37 on dist-test-slave-mrw4"
server_key: "2dbbb383bd119ea80bcbc38c18735255"
server_key_iv: "8def0e3846afa29bda07e4c1fa8055c3"
server_key_version: "encryptionkey@0"
I20260926 19:39:37.151641 7538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "50651ca54d1e4c81968582f079a7c380"
format_stamp: "Formatted at 2026-09-26 19:39:37 on dist-test-slave-mrw4"
server_key: "2dbbb383bd119ea80bcbc38c18735255"
server_key_iv: "8def0e3846afa29bda07e4c1fa8055c3"
server_key_version: "encryptionkey@0"
I20260926 19:39:37.156270 7538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 19:39:37.159438 7552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:37.160660 7538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:37.160797 7538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "50651ca54d1e4c81968582f079a7c380"
format_stamp: "Formatted at 2026-09-26 19:39:37 on dist-test-slave-mrw4"
server_key: "2dbbb383bd119ea80bcbc38c18735255"
server_key_iv: "8def0e3846afa29bda07e4c1fa8055c3"
server_key_version: "encryptionkey@0"
I20260926 19:39:37.161001 7538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:37.166082 7538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:37.166823 7538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:37.167042 7538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:37.167855 7538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:37.169070 7538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:37.169179 7538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:37.169278 7538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:37.169343 7538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:37.197000 7538 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:45591
I20260926 19:39:37.197073 7664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:45591 every 8 connection(s)
I20260926 19:39:37.198302 7538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:37.206063 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:37.200730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:38619 (local address 127.2.164.126:39687)
0926 19:39:37.200891 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:37.200895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:37.202292 (+ 1397us) server_negotiation.cc:407] Connection header received
0926 19:39:37.203921 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:37.203927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:37.203993 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:37.204079 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:37.205385 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:37.205391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:37.205396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:37.205413 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:37.205444 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:37.205455 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:37.205556 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:37.205731 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 19:39:37.205799 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:39:37.207012 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:37.201228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39687 (local address 127.2.164.66:38619)
0926 19:39:37.202123 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:37.202171 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:37.203642 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:37.204316 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:37.204324 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:37.204422 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:37.205064 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:39:37.205066 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:37.205104 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:39:37.205220 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:37.205536 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:37.205545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:37.205553 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 19:39:37.205831 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 19:39:37.206094 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":249,"threads_started":1}
I20260926 19:39:37.207563 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 7538
I20260926 19:39:37.207726 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:37.208251 2705 external_mini_cluster.cc:1428] Setting key 079199a9973bb48221e1e9a63259787f
I20260926 19:39:37.208527 7665 heartbeater.cc:347] Connected to a master server at 127.2.164.126:39687
I20260926 19:39:37.208840 7665 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:37.209534 7665 heartbeater.cc:511] Master 127.2.164.126:39687 requested a full tablet report, sending...
I20260926 19:39:37.210956 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39687
--builtin_ntp_servers=127.2.164.84:44345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 19:39:37.210995 7349 ts_manager.cc:194] Registered new tserver with Master: 50651ca54d1e4c81968582f079a7c380 (127.2.164.66:45591)
W20260926 19:39:37.511431 7668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:37.511694 7668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:37.511823 7668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:37.518779 7668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:37.518879 7668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:37.519016 7668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:37.527633 7668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:37.528800 7668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:37.530561 7668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:37.542537 7673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:37.542723 7676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:37.542959 7674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:37.543299 7668 server_base.cc:1034] running on GCE node
I20260926 19:39:37.543839 7668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:37.544659 7668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:37.545840 7668 hybrid_clock.cc:648] HybridClock initialized: now 1790451577545794 us; error 43 us; skew 500 ppm
I20260926 19:39:37.546124 7668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:39:37.548098 7668 webserver.cc:460] Webserver started at http://127.2.164.67:38423/ using document root <none> and password file <none>
I20260926 19:39:37.548704 7668 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:37.548797 7668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:37.549070 7668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:37.551671 7668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "301eed07b3d341bf8725c9e0e555c706"
format_stamp: "Formatted at 2026-09-26 19:39:37 on dist-test-slave-mrw4"
server_key: "7f72be0a2d8d2c0bd5c41f3da221ad96"
server_key_iv: "025b3d271ce1c9997ae18f4cb47e394b"
server_key_version: "encryptionkey@0"
I20260926 19:39:37.552382 7668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "301eed07b3d341bf8725c9e0e555c706"
format_stamp: "Formatted at 2026-09-26 19:39:37 on dist-test-slave-mrw4"
server_key: "7f72be0a2d8d2c0bd5c41f3da221ad96"
server_key_iv: "025b3d271ce1c9997ae18f4cb47e394b"
server_key_version: "encryptionkey@0"
I20260926 19:39:37.557061 7668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:39:37.560078 7682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:37.561308 7668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:39:37.561426 7668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "301eed07b3d341bf8725c9e0e555c706"
format_stamp: "Formatted at 2026-09-26 19:39:37 on dist-test-slave-mrw4"
server_key: "7f72be0a2d8d2c0bd5c41f3da221ad96"
server_key_iv: "025b3d271ce1c9997ae18f4cb47e394b"
server_key_version: "encryptionkey@0"
I20260926 19:39:37.561611 7668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:37.566751 7668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:37.567414 7668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:37.567639 7668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:37.568419 7668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:37.569687 7668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:37.569777 7668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:37.569872 7668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:37.569936 7668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:37.597047 7668 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:46267
I20260926 19:39:37.597093 7794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:46267 every 8 connection(s)
I20260926 19:39:37.598280 7668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:37.605873 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:37.600650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:38863 (local address 127.2.164.126:39687)
0926 19:39:37.600824 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:37.600829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:37.602163 (+ 1334us) server_negotiation.cc:407] Connection header received
0926 19:39:37.603769 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:37.603775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:37.603838 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:37.603922 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:37.605187 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:37.605193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:37.605198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:37.605218 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:37.605259 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:37.605271 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:37.605383 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:37.605549 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 19:39:37.605613 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:39:37.606863 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:37.601113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39687 (local address 127.2.164.67:38863)
0926 19:39:37.602005 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:37.602046 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:37.603493 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:37.604153 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:37.604161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:37.604253 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:37.604876 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:39:37.604879 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:37.604915 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:39:37.604996 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:37.605354 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:37.605365 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:37.605372 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 19:39:37.605619 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 19:39:37.605872 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":233,"threads_started":1}
I20260926 19:39:37.606885 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 7668
I20260926 19:39:37.607020 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:37.607481 2705 external_mini_cluster.cc:1428] Setting key 5558942007a70621ffee3517880b87bc
I20260926 19:39:37.608187 7795 heartbeater.cc:347] Connected to a master server at 127.2.164.126:39687
I20260926 19:39:37.608487 7795 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:37.609040 7795 heartbeater.cc:511] Master 127.2.164.126:39687 requested a full tablet report, sending...
I20260926 19:39:37.610520 7349 ts_manager.cc:194] Registered new tserver with Master: 301eed07b3d341bf8725c9e0e555c706 (127.2.164.67:46267)
I20260926 19:39:37.622231 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:39:37.628578 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:37.626242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.2.164.126:39687)
0926 19:39:37.626414 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:37.626421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:37.626972 (+ 551us) server_negotiation.cc:407] Connection header received
0926 19:39:37.627098 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:37.627101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:37.627149 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:37.627223 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:37.627993 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:37.627997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:37.628000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:37.628014 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:37.628043 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:37.628053 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:37.628142 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:37.628279 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 19:39:37.628322 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:39:37.632138 7349 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:39:37.638586 7349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:37.665587 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:37.650872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.2.164.66:45591)
0926 19:39:37.661342 (+ 10470us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:37.661352 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:37.662816 (+ 1464us) server_negotiation.cc:407] Connection header received
0926 19:39:37.662919 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:37.662927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:37.663139 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:37.663285 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:37.664808 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:37.664815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:37.664827 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:37.664854 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:37.664908 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:37.664927 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:37.665067 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:37.665133 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 19:39:37.665248 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10300,"thread_start_us":10170,"threads_started":1}
I20260926 19:39:37.665659 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:37.650440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.2.164.65:42959)
0926 19:39:37.661695 (+ 11255us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:37.661704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:37.662414 (+ 710us) server_negotiation.cc:407] Connection header received
0926 19:39:37.662488 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:37.662496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:37.662736 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:37.662880 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:37.664197 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:37.664205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:37.664569 (+ 364us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:37.664605 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:37.664671 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:37.664694 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:37.664825 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:37.665277 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 19:39:37.665354 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11120,"thread_start_us":11039,"threads_started":1}
I20260926 19:39:37.666188 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:37.649553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.2.164.67:46267)
0926 19:39:37.662435 (+ 12882us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:37.662443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:37.662468 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:39:37.662522 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:37.662528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:37.662736 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:37.662880 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:37.664968 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:37.664977 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:37.664989 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:37.665008 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:37.665064 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:37.665084 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:37.665269 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:37.665765 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 19:39:37.665850 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12725,"thread_start_us":12577,"threads_started":1}
I20260926 19:39:37.666522 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:37.649526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:42959 (local address 127.0.0.1:55114)
0926 19:39:37.662172 (+ 12646us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:37.662189 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:37.662297 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:37.662972 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:37.662979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:37.663076 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:37.663814 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:39:37.663818 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:37.663896 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:39:37.663995 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:37.664782 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:37.664791 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:37.664798 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 19:39:37.664951 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 19:39:37.665033 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12509,"thread_start_us":10741,"threads_started":1}
I20260926 19:39:37.666962 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:37.649869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:45591 (local address 127.0.0.1:51196)
0926 19:39:37.662708 (+ 12839us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:37.662721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:37.662813 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:37.663897 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:37.663901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:37.663917 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:37.664576 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:39:37.664578 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:37.664590 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:39:37.664647 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:37.665005 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:37.665010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:37.665014 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 19:39:37.665124 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 19:39:37.665287 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12737,"thread_start_us":11977,"threads_started":1}
I20260926 19:39:37.667068 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:37.649424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:46267 (local address 127.0.0.1:48944)
0926 19:39:37.660547 (+ 11123us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:37.660586 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:37.660714 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:37.663866 (+ 3152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:37.663870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:37.663892 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:37.664655 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:39:37.664658 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:37.664672 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:39:37.664710 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:37.665617 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:37.665620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:37.665624 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 19:39:37.666163 (+ 539us) client_negotiation.cc:240] Negotiation successful
0926 19:39:37.666217 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10836,"thread_start_us":13063,"threads_started":1}
I20260926 19:39:37.669888 7470 tablet_service.cc:1467] Processing CreateTablet for tablet e2924dc506b74b738ca095c2b5faca81 (DEFAULT_TABLE table=test-table [id=4cf31fcb60aa47488b4b1d0590e719c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:37.669886 7600 tablet_service.cc:1467] Processing CreateTablet for tablet e2924dc506b74b738ca095c2b5faca81 (DEFAULT_TABLE table=test-table [id=4cf31fcb60aa47488b4b1d0590e719c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:37.670987 7600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2924dc506b74b738ca095c2b5faca81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:37.671065 7470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2924dc506b74b738ca095c2b5faca81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:37.672748 7730 tablet_service.cc:1467] Processing CreateTablet for tablet e2924dc506b74b738ca095c2b5faca81 (DEFAULT_TABLE table=test-table [id=4cf31fcb60aa47488b4b1d0590e719c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:37.674002 7730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2924dc506b74b738ca095c2b5faca81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:37.681417 7813 tablet_bootstrap.cc:492] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: Bootstrap starting.
I20260926 19:39:37.684237 7814 tablet_bootstrap.cc:492] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a: Bootstrap starting.
I20260926 19:39:37.684585 7813 tablet_bootstrap.cc:654] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:37.685884 7813 log.cc:824] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:37.686446 7815 tablet_bootstrap.cc:492] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: Bootstrap starting.
I20260926 19:39:37.687824 7814 tablet_bootstrap.cc:654] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:37.688791 7814 log.cc:824] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:37.689631 7815 tablet_bootstrap.cc:654] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:37.689909 7813 tablet_bootstrap.cc:492] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: No bootstrap required, opened a new log
I20260926 19:39:37.690166 7813 ts_tablet_manager.cc:1397] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260926 19:39:37.690718 7815 log.cc:824] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:37.691107 7814 tablet_bootstrap.cc:492] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a: No bootstrap required, opened a new log
I20260926 19:39:37.691344 7814 ts_tablet_manager.cc:1397] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.003s
I20260926 19:39:37.693563 7815 tablet_bootstrap.cc:492] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: No bootstrap required, opened a new log
I20260926 19:39:37.693799 7815 ts_tablet_manager.cc:1397] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 19:39:37.698375 7814 raft_consensus.cc:348] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 } } peers { permanent_uuid: "827cd5fd73564465af06399391cbf36a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 42959 } } peers { permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 } }
I20260926 19:39:37.698716 7814 raft_consensus.cc:374] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:37.698836 7814 raft_consensus.cc:729] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827cd5fd73564465af06399391cbf36a, State: Initialized, Role: FOLLOWER
I20260926 19:39:37.699374 7814 consensus_queue.cc:260] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 } } peers { permanent_uuid: "827cd5fd73564465af06399391cbf36a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 42959 } } peers { permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 } }
I20260926 19:39:37.699556 7813 raft_consensus.cc:348] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 } } peers { permanent_uuid: "827cd5fd73564465af06399391cbf36a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 42959 } } peers { permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 } }
I20260926 19:39:37.699993 7813 raft_consensus.cc:374] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:37.700109 7813 raft_consensus.cc:729] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50651ca54d1e4c81968582f079a7c380, State: Initialized, Role: FOLLOWER
I20260926 19:39:37.700408 7535 heartbeater.cc:503] Master 127.2.164.126:39687 was elected leader, sending a full tablet report...
I20260926 19:39:37.700678 7813 consensus_queue.cc:260] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 } } peers { permanent_uuid: "827cd5fd73564465af06399391cbf36a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 42959 } } peers { permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 } }
I20260926 19:39:37.700945 7814 ts_tablet_manager.cc:1428] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20260926 19:39:37.702045 7665 heartbeater.cc:503] Master 127.2.164.126:39687 was elected leader, sending a full tablet report...
I20260926 19:39:37.702517 7813 ts_tablet_manager.cc:1428] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260926 19:39:37.703261 7815 raft_consensus.cc:348] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 } } peers { permanent_uuid: "827cd5fd73564465af06399391cbf36a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 42959 } } peers { permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 } }
I20260926 19:39:37.703608 7815 raft_consensus.cc:374] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:37.703708 7815 raft_consensus.cc:729] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 301eed07b3d341bf8725c9e0e555c706, State: Initialized, Role: FOLLOWER
I20260926 19:39:37.704268 7815 consensus_queue.cc:260] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 } } peers { permanent_uuid: "827cd5fd73564465af06399391cbf36a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 42959 } } peers { permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 } }
I20260926 19:39:37.705375 7795 heartbeater.cc:503] Master 127.2.164.126:39687 was elected leader, sending a full tablet report...
I20260926 19:39:37.705927 7815 ts_tablet_manager.cc:1428] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
W20260926 19:39:37.770071 7536 tablet.cc:2369] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:39:37.849767 7796 tablet.cc:2369] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:39:37.950093 7666 tablet.cc:2369] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:38.038380 7819 raft_consensus.cc:482] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:38.038689 7819 raft_consensus.cc:504] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 } } peers { permanent_uuid: "827cd5fd73564465af06399391cbf36a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 42959 } } peers { permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 } }
I20260926 19:39:38.040377 7819 leader_election.cc:288] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 301eed07b3d341bf8725c9e0e555c706 (127.2.164.67:46267), 50651ca54d1e4c81968582f079a7c380 (127.2.164.66:45591)
I20260926 19:39:38.043321 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.040867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:41207 (local address 127.2.164.66:45591)
0926 19:39:38.041023 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:38.041029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:38.041492 (+ 463us) server_negotiation.cc:407] Connection header received
0926 19:39:38.041660 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:38.041663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:38.041721 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:38.041797 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:38.042723 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:38.042730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:38.042734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:38.042754 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:38.042790 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:38.042806 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:38.042913 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:38.043001 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 19:39:38.043057 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:39:38.043452 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.040799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:35629 (local address 127.2.164.67:46267)
0926 19:39:38.040989 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:38.040997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:38.041860 (+ 863us) server_negotiation.cc:407] Connection header received
0926 19:39:38.041990 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:38.041996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:38.042047 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:38.042123 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:38.042896 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:38.042901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:38.042906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:38.042925 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:38.042962 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:38.042978 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:38.043085 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:38.043177 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 19:39:38.043233 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:39:38.043665 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.040723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:45591 (local address 127.2.164.65:41207)
0926 19:39:38.041379 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:38.041395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:38.041520 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:38.041839 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:38.041843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:38.041866 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:38.042499 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:39:38.042503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:38.042521 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:39:38.042569 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:38.042899 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:38.042903 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:38.042908 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:39:38.043037 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 19:39:38.043087 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":287,"threads_started":1}
I20260926 19:39:38.043900 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.040729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:46267 (local address 127.2.164.65:35629)
0926 19:39:38.041764 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:38.041778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:38.041878 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:38.042157 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:38.042162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:38.042183 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:38.042692 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:39:38.042695 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:38.042709 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:39:38.042758 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:38.043069 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:38.043074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:38.043079 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:39:38.043245 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 19:39:38.043290 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":306,"thread_start_us":417,"threads_started":1}
I20260926 19:39:38.044508 7620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2924dc506b74b738ca095c2b5faca81" candidate_uuid: "827cd5fd73564465af06399391cbf36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50651ca54d1e4c81968582f079a7c380" is_pre_election: true
I20260926 19:39:38.044693 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2924dc506b74b738ca095c2b5faca81" candidate_uuid: "827cd5fd73564465af06399391cbf36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "301eed07b3d341bf8725c9e0e555c706" is_pre_election: true
I20260926 19:39:38.044952 7620 raft_consensus.cc:2454] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827cd5fd73564465af06399391cbf36a in term 0.
I20260926 19:39:38.045128 7820 raft_consensus.cc:482] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:38.045156 7750 raft_consensus.cc:2454] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827cd5fd73564465af06399391cbf36a in term 0.
I20260926 19:39:38.045353 7820 raft_consensus.cc:504] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 } } peers { permanent_uuid: "827cd5fd73564465af06399391cbf36a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 42959 } } peers { permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 } }
I20260926 19:39:38.045598 7426 leader_election.cc:302] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50651ca54d1e4c81968582f079a7c380, 827cd5fd73564465af06399391cbf36a; no voters:
I20260926 19:39:38.045994 7819 raft_consensus.cc:2781] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:38.046115 7819 raft_consensus.cc:482] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:38.046202 7819 raft_consensus.cc:3037] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:38.046869 7820 leader_election.cc:288] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 301eed07b3d341bf8725c9e0e555c706 (127.2.164.67:46267), 827cd5fd73564465af06399391cbf36a (127.2.164.65:42959)
I20260926 19:39:38.048695 7819 raft_consensus.cc:504] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 } } peers { permanent_uuid: "827cd5fd73564465af06399391cbf36a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 42959 } } peers { permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 } }
I20260926 19:39:38.049348 7819 leader_election.cc:288] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [CANDIDATE]: Term 1 election: Requested vote from peers 301eed07b3d341bf8725c9e0e555c706 (127.2.164.67:46267), 50651ca54d1e4c81968582f079a7c380 (127.2.164.66:45591)
I20260926 19:39:38.049751 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2924dc506b74b738ca095c2b5faca81" candidate_uuid: "827cd5fd73564465af06399391cbf36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "301eed07b3d341bf8725c9e0e555c706"
I20260926 19:39:38.049950 7750 raft_consensus.cc:3037] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:38.050038 7620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2924dc506b74b738ca095c2b5faca81" candidate_uuid: "827cd5fd73564465af06399391cbf36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50651ca54d1e4c81968582f079a7c380"
I20260926 19:39:38.050236 7620 raft_consensus.cc:3037] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:38.050670 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.047069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:46267 (local address 127.2.164.66:37595)
0926 19:39:38.047630 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:38.047653 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:38.047886 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:38.048344 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:38.048349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:38.048374 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:38.049071 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:39:38.049075 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:38.049093 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:39:38.049142 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:38.049619 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:38.049628 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:38.049631 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 19:39:38.050025 (+ 394us) client_negotiation.cc:240] Negotiation successful
0926 19:39:38.050094 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":2505,"threads_started":1}
I20260926 19:39:38.051209 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.047365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:37595 (local address 127.2.164.67:46267)
0926 19:39:38.047506 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:38.047511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:38.047792 (+ 281us) server_negotiation.cc:407] Connection header received
0926 19:39:38.048017 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:38.048023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:38.048086 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:38.048148 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:38.049325 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:38.049331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:38.049335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:38.049351 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:38.049381 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:38.049395 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:38.049497 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:38.049767 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 19:39:38.049821 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:39:38.051949 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.047573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:35553 (local address 127.2.164.65:42959)
0926 19:39:38.047735 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:38.047741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:38.049945 (+ 2204us) server_negotiation.cc:407] Connection header received
0926 19:39:38.050139 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:38.050143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:38.050193 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:38.050256 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:38.051337 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:38.051342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:38.051347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:38.051364 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:38.051397 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:38.051411 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:38.051517 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:38.051657 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 19:39:38.051711 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:39:38.052579 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.047119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:42959 (local address 127.2.164.66:35553)
0926 19:39:38.049854 (+ 2735us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:38.049876 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:38.050010 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:38.050435 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:38.050439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:38.050465 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:38.051131 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:39:38.051138 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:38.051153 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:39:38.051198 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:38.051511 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:38.051516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:38.051521 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:39:38.052151 (+ 630us) client_negotiation.cc:240] Negotiation successful
0926 19:39:38.052199 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2625,"thread_start_us":670,"threads_started":1}
I20260926 19:39:38.052848 7620 raft_consensus.cc:2454] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827cd5fd73564465af06399391cbf36a in term 1.
I20260926 19:39:38.053215 7426 leader_election.cc:302] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50651ca54d1e4c81968582f079a7c380, 827cd5fd73564465af06399391cbf36a; no voters:
I20260926 19:39:38.053458 7819 raft_consensus.cc:2781] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:38.053468 7750 raft_consensus.cc:2454] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827cd5fd73564465af06399391cbf36a in term 1.
I20260926 19:39:38.053851 7749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2924dc506b74b738ca095c2b5faca81" candidate_uuid: "50651ca54d1e4c81968582f079a7c380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "301eed07b3d341bf8725c9e0e555c706" is_pre_election: true
I20260926 19:39:38.054216 7749 raft_consensus.cc:2379] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 50651ca54d1e4c81968582f079a7c380 in current term 1: Already voted for candidate 827cd5fd73564465af06399391cbf36a in this term.
I20260926 19:39:38.054425 7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2924dc506b74b738ca095c2b5faca81" candidate_uuid: "50651ca54d1e4c81968582f079a7c380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827cd5fd73564465af06399391cbf36a" is_pre_election: true
I20260926 19:39:38.054697 7819 raft_consensus.cc:686] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 1 LEADER]: Becoming Leader. State: Replica: 827cd5fd73564465af06399391cbf36a, State: Running, Role: LEADER
I20260926 19:39:38.055099 7819 consensus_queue.cc:237] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 } } peers { permanent_uuid: "827cd5fd73564465af06399391cbf36a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 42959 } } peers { permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 } }
I20260926 19:39:38.058454 7553 leader_election.cc:302] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50651ca54d1e4c81968582f079a7c380; no voters: 301eed07b3d341bf8725c9e0e555c706, 827cd5fd73564465af06399391cbf36a
I20260926 19:39:38.058768 7820 raft_consensus.cc:2726] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 19:39:38.059845 7349 catalog_manager.cc:5495] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a reported cstate change: term changed from 0 to 1, leader changed from <none> to 827cd5fd73564465af06399391cbf36a (127.2.164.65). New cstate: current_term: 1 leader_uuid: "827cd5fd73564465af06399391cbf36a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "827cd5fd73564465af06399391cbf36a" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 42959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 } health_report { overall_health: UNKNOWN } } }
I20260926 19:39:38.106132 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.104156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.2.164.65:42959)
0926 19:39:38.104345 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:38.104350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:38.104435 (+ 85us) server_negotiation.cc:407] Connection header received
0926 19:39:38.104579 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:38.104583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:38.104675 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:38.104761 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:38.105551 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:38.105556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:38.105559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:38.105574 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:38.105626 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:38.105637 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:38.105735 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:38.105834 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 19:39:38.105878 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:39:38.113730 7749 raft_consensus.cc:1261] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706 [term 1 FOLLOWER]: Refusing update from remote peer 827cd5fd73564465af06399391cbf36a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:38.113791 7620 raft_consensus.cc:1261] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 1 FOLLOWER]: Refusing update from remote peer 827cd5fd73564465af06399391cbf36a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:38.114547 7826 consensus_queue.cc:1035] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [LEADER]: Connected to new peer: Peer: permanent_uuid: "301eed07b3d341bf8725c9e0e555c706" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:38.114854 7819 consensus_queue.cc:1035] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50651ca54d1e4c81968582f079a7c380" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:38.125303 7829 mvcc.cc:205] Tried to move back new op lower bound from 7333689663944876032 to 7333689663717105664. Current Snapshot: MvccSnapshot[applied={T|T < 7333689663944876032}]
I20260926 19:39:38.129413 7831 mvcc.cc:205] Tried to move back new op lower bound from 7333689663944876032 to 7333689663717105664. Current Snapshot: MvccSnapshot[applied={T|T < 7333689663944876032}]
I20260926 19:39:38.138512 7349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:38.138737 7349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:38.141976 7349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eb19e72970146c7a86bce84ae802b52: Sending DeleteTablet for 3 replicas of tablet e2924dc506b74b738ca095c2b5faca81
I20260926 19:39:38.142896 7730 tablet_service.cc:1514] Processing DeleteTablet for tablet e2924dc506b74b738ca095c2b5faca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:38 UTC) from {username='slave'} at 127.0.0.1:48944
I20260926 19:39:38.142997 7470 tablet_service.cc:1514] Processing DeleteTablet for tablet e2924dc506b74b738ca095c2b5faca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:38 UTC) from {username='slave'} at 127.0.0.1:55114
I20260926 19:39:38.143883 7600 tablet_service.cc:1514] Processing DeleteTablet for tablet e2924dc506b74b738ca095c2b5faca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:38 UTC) from {username='slave'} at 127.0.0.1:51196
I20260926 19:39:38.144353 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 7408
I20260926 19:39:38.145184 7840 tablet_replica.cc:330] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a: stopping tablet replica
I20260926 19:39:38.145599 7840 raft_consensus.cc:2229] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:39:38.146188 7840 raft_consensus.cc:2258] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:38.146889 7839 tablet_replica.cc:330] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: stopping tablet replica
I20260926 19:39:38.147303 7839 raft_consensus.cc:2229] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:38.147790 7839 raft_consensus.cc:2258] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:38.148494 7841 tablet_replica.cc:330] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: stopping tablet replica
I20260926 19:39:38.148769 7841 raft_consensus.cc:2229] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:38.149099 7841 raft_consensus.cc:2258] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:38.150880 7841 ts_tablet_manager.cc:1905] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:38.151273 7840 ts_tablet_manager.cc:1905] T e2924dc506b74b738ca095c2b5faca81 P 827cd5fd73564465af06399391cbf36a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:38.152077 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.146217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.2.164.65:42959)
0926 19:39:38.146389 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:38.146395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:38.149828 (+ 3433us) server_negotiation.cc:407] Connection header received
0926 19:39:38.150095 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:38.150100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:38.150157 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:38.150224 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:38.151362 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:38.151367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:38.151371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:38.151389 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:38.151422 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:38.151436 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:38.151521 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:38.151756 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 19:39:38.151815 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:39:38.153524 7839 ts_tablet_manager.cc:1905] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:38.156555 7530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:38.157430 7841 ts_tablet_manager.cc:1918] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:38.157598 7841 log.cc:1196] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/e2924dc506b74b738ca095c2b5faca81
I20260926 19:39:38.158113 7841 ts_tablet_manager.cc:1939] T e2924dc506b74b738ca095c2b5faca81 P 50651ca54d1e4c81968582f079a7c380: Deleting consensus metadata
I20260926 19:39:38.159775 7336 catalog_manager.cc:4841] TS 50651ca54d1e4c81968582f079a7c380 (127.2.164.66:45591): tablet e2924dc506b74b738ca095c2b5faca81 (table test-table [id=4cf31fcb60aa47488b4b1d0590e719c6]) successfully deleted
I20260926 19:39:38.159999 7839 ts_tablet_manager.cc:1918] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:38.160125 7839 log.cc:1196] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/e2924dc506b74b738ca095c2b5faca81
I20260926 19:39:38.160549 7839 ts_tablet_manager.cc:1939] T e2924dc506b74b738ca095c2b5faca81 P 301eed07b3d341bf8725c9e0e555c706: Deleting consensus metadata
I20260926 19:39:38.161897 7335 catalog_manager.cc:4841] TS 301eed07b3d341bf8725c9e0e555c706 (127.2.164.67:46267): tablet e2924dc506b74b738ca095c2b5faca81 (table test-table [id=4cf31fcb60aa47488b4b1d0590e719c6]) successfully deleted
I20260926 19:39:38.262359 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 7408
W20260926 19:39:38.275758 7333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:42959 (error 108)
I20260926 19:39:38.276515 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 7538
I20260926 19:39:38.278084 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.277090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:42959 (local address 127.0.0.1:55136)
0926 19:39:38.277820 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:38.277916 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":525,"threads_started":1}
W20260926 19:39:38.278478 7333 catalog_manager.cc:4584] TS 827cd5fd73564465af06399391cbf36a (127.2.164.65:42959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2924dc506b74b738ca095c2b5faca81: Network error: Client connection negotiation failed: client connection to 127.2.164.65:42959: connect: Connection refused (error 111)
I20260926 19:39:38.282976 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.277262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.2.164.66:45591)
0926 19:39:38.277466 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:38.277473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:38.277717 (+ 244us) server_negotiation.cc:407] Connection header received
0926 19:39:38.277785 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:38.277790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:38.277916 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:38.278008 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:38.282418 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:38.282422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:38.282426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:38.282443 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:38.282473 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:38.282485 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:38.282592 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:38.282669 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 19:39:38.282715 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 19:39:38.283567 7660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:38.331988 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.331468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:42959 (local address 127.0.0.1:55144)
0926 19:39:38.331702 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:38.331825 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 19:39:38.332999 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.332629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:42959 (local address 127.0.0.1:55148)
0926 19:39:38.332765 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:38.332847 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 19:39:38.369086 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 7538
I20260926 19:39:38.381927 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 7668
I20260926 19:39:38.384338 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.382587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.2.164.67:46267)
0926 19:39:38.382743 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:38.382748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:38.382822 (+ 74us) server_negotiation.cc:407] Connection header received
0926 19:39:38.382981 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:38.382985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:38.383061 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:38.383136 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:38.383888 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:38.383892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:38.383895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:38.383909 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:38.383939 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:38.383950 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:38.384039 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:38.384105 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 19:39:38.384148 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:39:38.384909 7790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:38.393581 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.393082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:42959 (local address 127.0.0.1:55162)
0926 19:39:38.393314 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:38.393414 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 19:39:38.394577 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.394182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:42959 (local address 127.0.0.1:55178)
0926 19:39:38.394332 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:38.394412 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 19:39:38.468079 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 7668
I20260926 19:39:38.475101 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.474609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:42959 (local address 127.0.0.1:55184)
0926 19:39:38.474823 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:38.474937 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 19:39:38.476199 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:38.475758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:42959 (local address 127.0.0.1:55190)
0926 19:39:38.475951 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:38.476049 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 19:39:38.480959 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 7318
I20260926 19:39:38.481921 7379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:38.582721 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 7318
2026-09-26T19:39:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2734 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 19:39:38.618721 2705 mini_oidc.cc:152] Starting JWKS server
I20260926 19:39:38.621125 2705 webserver.cc:460] Webserver started at https://127.0.0.1:32881/ using document root <none> and password file <none>
I20260926 19:39:38.621447 2705 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 19:39:38.622014 2705 webserver.cc:460] Webserver started at http://127.0.0.1:38445/ using document root <none> and password file <none>
2026-09-26T19:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:38Z Disabled control of system clock
I20260926 19:39:38.644320 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40937
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:41475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:40937
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32881/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/testchainca.cert with env {}
W20260926 19:39:38.941859 7854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:38.942137 7854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:38.942221 7854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:38.949236 7854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:38.949334 7854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:38.949375 7854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:38.949431 7854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:38.949473 7854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 19:39:38.949532 7854 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 19:39:38.949568 7854 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32881/jwks.json
I20260926 19:39:38.959070 7854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:40937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32881/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:38.960264 7854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:38.961459 7854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:38.972139 7862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:38.972191 7860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:38.972223 7859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:38.972790 7854 server_base.cc:1034] running on GCE node
I20260926 19:39:38.973528 7854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:38.974818 7854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:38.976020 7854 hybrid_clock.cc:648] HybridClock initialized: now 1790451578975967 us; error 42 us; skew 500 ppm
I20260926 19:39:38.976346 7854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:39:38.978475 7854 webserver.cc:460] Webserver started at http://127.2.164.126:44447/ using document root <none> and password file <none>
I20260926 19:39:38.979111 7854 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:38.979197 7854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:38.979480 7854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:38.982226 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "fcfe7bd0c766441fb58d1bcbb4c87e0d"
format_stamp: "Formatted at 2026-09-26 19:39:38 on dist-test-slave-mrw4"
server_key: "9fe1d4d89d694eb2fbe7199b40b88637"
server_key_iv: "bfaee7c60948fbfdd76fccaa38dff3d3"
server_key_version: "encryptionkey@0"
I20260926 19:39:38.982942 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "fcfe7bd0c766441fb58d1bcbb4c87e0d"
format_stamp: "Formatted at 2026-09-26 19:39:38 on dist-test-slave-mrw4"
server_key: "9fe1d4d89d694eb2fbe7199b40b88637"
server_key_iv: "bfaee7c60948fbfdd76fccaa38dff3d3"
server_key_version: "encryptionkey@0"
I20260926 19:39:38.987682 7854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 19:39:38.990883 7868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:38.992132 7854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:39:38.992273 7854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "fcfe7bd0c766441fb58d1bcbb4c87e0d"
format_stamp: "Formatted at 2026-09-26 19:39:38 on dist-test-slave-mrw4"
server_key: "9fe1d4d89d694eb2fbe7199b40b88637"
server_key_iv: "bfaee7c60948fbfdd76fccaa38dff3d3"
server_key_version: "encryptionkey@0"
I20260926 19:39:38.992447 7854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:39.034226 7854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:39.035081 7854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:39.035372 7854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:39.052271 7854 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:40937
I20260926 19:39:39.052278 7919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:40937 every 8 connection(s)
I20260926 19:39:39.053568 7854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:39.057060 7920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:39.059103 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 7854
I20260926 19:39:39.059315 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:39.060024 2705 external_mini_cluster.cc:1428] Setting key b5cbfef2b7436498d1cd33b16a92ac1d
I20260926 19:39:39.068228 7920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d: Bootstrap starting.
I20260926 19:39:39.071287 7920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:39.072299 7920 log.cc:824] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:39.072505 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:39.061896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.2.164.126:40937)
0926 19:39:39.062804 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:39.062818 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:39.062858 (+ 40us) server_negotiation.cc:407] Connection header received
0926 19:39:39.064459 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:39.064480 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:39.064913 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:39.065301 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:39.065835 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.067108 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.068411 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.069045 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.069225 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:39.070297 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:39.070305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:39.070317 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:39.070834 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:39.070907 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:39.070927 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:39.071033 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:39.071253 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 19:39:39.071581 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":287,"threads_started":1}
I20260926 19:39:39.074903 7920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d: No bootstrap required, opened a new log
I20260926 19:39:39.075935 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:39:39.081944 7920 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe7bd0c766441fb58d1bcbb4c87e0d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40937 } }
I20260926 19:39:39.082283 7920 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:39.082370 7920 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfe7bd0c766441fb58d1bcbb4c87e0d, State: Initialized, Role: FOLLOWER
I20260926 19:39:39.082904 7920 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe7bd0c766441fb58d1bcbb4c87e0d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40937 } }
I20260926 19:39:39.083101 7920 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:39.083216 7920 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:39.083359 7920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:39.085593 7920 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe7bd0c766441fb58d1bcbb4c87e0d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40937 } }
I20260926 19:39:39.086040 7920 leader_election.cc:302] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcfe7bd0c766441fb58d1bcbb4c87e0d; no voters:
I20260926 19:39:39.086511 7920 leader_election.cc:288] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:39.086620 7925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:39.087473 7925 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [term 1 LEADER]: Becoming Leader. State: Replica: fcfe7bd0c766441fb58d1bcbb4c87e0d, State: Running, Role: LEADER
I20260926 19:39:39.087994 7925 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe7bd0c766441fb58d1bcbb4c87e0d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40937 } }
I20260926 19:39:39.088514 7920 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:39.090798 7926 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcfe7bd0c766441fb58d1bcbb4c87e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe7bd0c766441fb58d1bcbb4c87e0d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40937 } } }
I20260926 19:39:39.090768 7927 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcfe7bd0c766441fb58d1bcbb4c87e0d. Latest consensus state: current_term: 1 leader_uuid: "fcfe7bd0c766441fb58d1bcbb4c87e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe7bd0c766441fb58d1bcbb4c87e0d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40937 } } }
I20260926 19:39:39.091156 7926 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:39.091161 7927 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:39.092875 7933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:39.100731 7933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:39.108129 7933 catalog_manager.cc:1347] Generated new cluster ID: 6f0b78330d10482ca05be70ab5f7480d
I20260926 19:39:39.108227 7933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:39.126996 7933 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:39.128260 7933 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:39.134842 7933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcfe7bd0c766441fb58d1bcbb4c87e0d: Generated new TSK 0
I20260926 19:39:39.135572 7933 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 19:39:39.177631 7950 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 19:39:39.177796 7950 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 19:39:39.178671 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:39.157935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.2.164.126:40937)
0926 19:39:39.158150 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:39.158156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:39.158601 (+ 445us) server_negotiation.cc:407] Connection header received
0926 19:39:39.158908 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:39.158912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:39.158998 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:39.159113 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0926 19:39:39.159707 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.160503 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.162512 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.162967 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.163070 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:39.163682 (+ 612us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 19:39:39.178186 (+ 14504us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 19:39:39.178452 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 19:39:39.178794 7923 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 19:39:39.180549 7949 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:39:39.157846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40937 (local address 127.0.0.1:52480)
0926 19:39:39.158496 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:39.158515 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:39.158690 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:39.159165 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:39.159170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:39.159184 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 19:39:39.159553 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:39.159566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.160627 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.160633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:39.162350 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:39.162365 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.163147 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.163154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:39.163349 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:39.163421 (+ 72us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 19:39:39.178452 (+ 15031us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 19:39:39.178801 (+ 349us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.164.126:40937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":279,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.164.126:40937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 19:39:39.182215 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 7854
I20260926 19:39:39.183380 7915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:39.274351 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 7854
2026-09-26T19:39:39Z chronyd exiting
I20260926 19:39:39.320632 2705 test_util.cc:175] -----------------------------------------------
I20260926 19:39:39.320768 2705 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790451540687502-2705-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (706 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 19:39:39.324919 2705 mini_oidc.cc:152] Starting JWKS server
I20260926 19:39:39.327769 2705 webserver.cc:460] Webserver started at https://127.0.0.1:43739/ using document root <none> and password file <none>
I20260926 19:39:39.328006 2705 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 19:39:39.328742 2705 webserver.cc:460] Webserver started at http://127.0.0.1:34355/ using document root <none> and password file <none>
2026-09-26T19:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:39Z Disabled control of system clock
I20260926 19:39:39.350438 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34493
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:34493
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43739/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/testchainca.cert with env {}
W20260926 19:39:39.647230 7957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:39.647496 7957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:39.647588 7957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:39.654523 7957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:39.654616 7957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:39.654652 7957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:39.654706 7957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:39.654747 7957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 19:39:39.654811 7957 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 19:39:39.654850 7957 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43739/jwks.json
I20260926 19:39:39.664274 7957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:34493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43739/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:39.665551 7957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:39.666720 7957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:39.677296 7962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:39.677349 7965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:39.677379 7963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:39.678063 7957 server_base.cc:1034] running on GCE node
I20260926 19:39:39.678740 7957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:39.679979 7957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:39.681181 7957 hybrid_clock.cc:648] HybridClock initialized: now 1790451579681127 us; error 29 us; skew 500 ppm
I20260926 19:39:39.681484 7957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:39:39.683460 7957 webserver.cc:460] Webserver started at http://127.2.164.126:46813/ using document root <none> and password file <none>
I20260926 19:39:39.684073 7957 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:39.684165 7957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:39.684449 7957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:39.687088 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "3077bc94f3e149a4bcd61d45ff9139b2"
format_stamp: "Formatted at 2026-09-26 19:39:39 on dist-test-slave-mrw4"
server_key: "24ae99e449cf85b73f0be59b528a1e42"
server_key_iv: "61bc4f79b862b1437112aa4b64f238e1"
server_key_version: "encryptionkey@0"
I20260926 19:39:39.687813 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "3077bc94f3e149a4bcd61d45ff9139b2"
format_stamp: "Formatted at 2026-09-26 19:39:39 on dist-test-slave-mrw4"
server_key: "24ae99e449cf85b73f0be59b528a1e42"
server_key_iv: "61bc4f79b862b1437112aa4b64f238e1"
server_key_version: "encryptionkey@0"
I20260926 19:39:39.692315 7957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 19:39:39.695477 7971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:39.696712 7957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:39:39.696862 7957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "3077bc94f3e149a4bcd61d45ff9139b2"
format_stamp: "Formatted at 2026-09-26 19:39:39 on dist-test-slave-mrw4"
server_key: "24ae99e449cf85b73f0be59b528a1e42"
server_key_iv: "61bc4f79b862b1437112aa4b64f238e1"
server_key_version: "encryptionkey@0"
I20260926 19:39:39.697038 7957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:39.709545 7957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:39.710261 7957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:39.710517 7957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:39.726330 7957 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:34493
I20260926 19:39:39.726354 8022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:34493 every 8 connection(s)
I20260926 19:39:39.727499 7957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:39.731081 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:39.733557 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 7957
I20260926 19:39:39.733738 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:39.734526 2705 external_mini_cluster.cc:1428] Setting key 0e84b3ce63e5af9d1521cfb178a03468
I20260926 19:39:39.742316 8023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2: Bootstrap starting.
I20260926 19:39:39.745616 8023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:39.746765 8023 log.cc:824] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:39.747562 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:39.736299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.2.164.126:34493)
0926 19:39:39.737197 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:39.737212 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:39.737252 (+ 40us) server_negotiation.cc:407] Connection header received
0926 19:39:39.738796 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:39.738815 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:39.739251 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:39.739561 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:39.740048 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.741350 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.743032 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.743706 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.743838 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:39.744987 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:39.744993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:39.745005 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:39.745533 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:39.745624 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:39.745643 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:39.745779 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:39.746044 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 19:39:39.746338 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":319,"threads_started":1}
I20260926 19:39:39.749444 8023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2: No bootstrap required, opened a new log
I20260926 19:39:39.751152 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:39:39.757757 8023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3077bc94f3e149a4bcd61d45ff9139b2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34493 } }
I20260926 19:39:39.758055 8023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:39.758127 8023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3077bc94f3e149a4bcd61d45ff9139b2, State: Initialized, Role: FOLLOWER
I20260926 19:39:39.758613 8023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3077bc94f3e149a4bcd61d45ff9139b2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34493 } }
I20260926 19:39:39.758808 8023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:39.758901 8023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:39.759008 8023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:39.760900 8023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3077bc94f3e149a4bcd61d45ff9139b2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34493 } }
I20260926 19:39:39.761409 8023 leader_election.cc:302] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3077bc94f3e149a4bcd61d45ff9139b2; no voters:
I20260926 19:39:39.761895 8023 leader_election.cc:288] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:39.762020 8028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:39.762600 8028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [term 1 LEADER]: Becoming Leader. State: Replica: 3077bc94f3e149a4bcd61d45ff9139b2, State: Running, Role: LEADER
I20260926 19:39:39.763054 8028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3077bc94f3e149a4bcd61d45ff9139b2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34493 } }
I20260926 19:39:39.763628 8023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:39.765738 8030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3077bc94f3e149a4bcd61d45ff9139b2. Latest consensus state: current_term: 1 leader_uuid: "3077bc94f3e149a4bcd61d45ff9139b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3077bc94f3e149a4bcd61d45ff9139b2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34493 } } }
I20260926 19:39:39.766147 8030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:39.767409 8029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3077bc94f3e149a4bcd61d45ff9139b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3077bc94f3e149a4bcd61d45ff9139b2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34493 } } }
I20260926 19:39:39.767759 8029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:39.768988 8038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:39.773908 8038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:39.782059 8038 catalog_manager.cc:1347] Generated new cluster ID: 9e22e99112d546c39290580d54d89b07
I20260926 19:39:39.782148 8038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:39.811261 8038 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:39.812471 8038 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:39.822432 8038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3077bc94f3e149a4bcd61d45ff9139b2: Generated new TSK 0
I20260926 19:39:39.823390 8038 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 19:39:39.855101 8053 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 19:39:39.855211 8053 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 19:39:39.856063 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:39.837027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.2.164.126:34493)
0926 19:39:39.837243 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:39.837250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:39.837658 (+ 408us) server_negotiation.cc:407] Connection header received
0926 19:39:39.837874 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:39.837877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:39.837938 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:39.838052 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0926 19:39:39.838545 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.839247 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.841210 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.841620 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.841711 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:39.842203 (+ 492us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 19:39:39.855617 (+ 13414us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 19:39:39.855874 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 19:39:39.856166 8026 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 19:39:39.856417 8052 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:39:39.836925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34493 (local address 127.0.0.1:50460)
0926 19:39:39.837555 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:39.837580 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:39.837718 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:39.838078 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:39.838083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:39.838096 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 19:39:39.838393 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:39.838405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.839365 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.839371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:39.841043 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:39.841058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:39.841772 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:39.841778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:39.841947 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:39.841991 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 19:39:39.855805 (+ 13814us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 19:39:39.856036 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.164.126:34493: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":346,"threads_started":1}
I20260926 19:39:39.857398 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 7957
I20260926 19:39:39.858373 8018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:39.948884 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 7957
2026-09-26T19:39:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (675 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 19:39:40.000128 2705 mini_oidc.cc:152] Starting JWKS server
I20260926 19:39:40.001942 2705 webserver.cc:460] Webserver started at https://127.0.0.1:46111/ using document root <none> and password file <none>
I20260926 19:39:40.002166 2705 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 19:39:40.002743 2705 webserver.cc:460] Webserver started at http://127.0.0.1:43783/ using document root <none> and password file <none>
2026-09-26T19:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:40Z Disabled control of system clock
I20260926 19:39:40.024339 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44301
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:44301
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46111/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 19:39:40.320096 8060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:40.320360 8060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:40.320478 8060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:40.327324 8060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:40.327411 8060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:40.327467 8060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:40.327539 8060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:40.327594 8060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 19:39:40.327689 8060 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 19:39:40.327735 8060 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46111/jwks.json
I20260926 19:39:40.336304 8060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:44301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46111/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:40.337528 8060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:40.339222 8060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:40.349787 8065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:40.349776 8066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:40.349879 8068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:40.350363 8060 server_base.cc:1034] running on GCE node
I20260926 19:39:40.351105 8060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:40.352366 8060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:40.353555 8060 hybrid_clock.cc:648] HybridClock initialized: now 1790451580353503 us; error 36 us; skew 500 ppm
I20260926 19:39:40.353858 8060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:39:40.355891 8060 webserver.cc:460] Webserver started at http://127.2.164.126:34801/ using document root <none> and password file <none>
I20260926 19:39:40.356508 8060 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:40.356603 8060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:40.356894 8060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:40.359553 8060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "d09e803e92384c849eecb2bbef6872e6"
format_stamp: "Formatted at 2026-09-26 19:39:40 on dist-test-slave-mrw4"
server_key: "76f40c2971e5c6bc1bdd727e57414e06"
server_key_iv: "ac2af93f7e4289dd99d2a3b2499a05c9"
server_key_version: "encryptionkey@0"
I20260926 19:39:40.360275 8060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "d09e803e92384c849eecb2bbef6872e6"
format_stamp: "Formatted at 2026-09-26 19:39:40 on dist-test-slave-mrw4"
server_key: "76f40c2971e5c6bc1bdd727e57414e06"
server_key_iv: "ac2af93f7e4289dd99d2a3b2499a05c9"
server_key_version: "encryptionkey@0"
I20260926 19:39:40.364837 8060 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 19:39:40.367929 8074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:40.369105 8060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:40.369292 8060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "d09e803e92384c849eecb2bbef6872e6"
format_stamp: "Formatted at 2026-09-26 19:39:40 on dist-test-slave-mrw4"
server_key: "76f40c2971e5c6bc1bdd727e57414e06"
server_key_iv: "ac2af93f7e4289dd99d2a3b2499a05c9"
server_key_version: "encryptionkey@0"
I20260926 19:39:40.369468 8060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:40.390542 8060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:40.391285 8060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:40.391552 8060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:40.407413 8060 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:44301
I20260926 19:39:40.407434 8125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:44301 every 8 connection(s)
I20260926 19:39:40.408581 8060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:40.411937 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:40.418376 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 8060
I20260926 19:39:40.418572 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:40.419324 2705 external_mini_cluster.cc:1428] Setting key 5cde26035bcfec9631f758547d6b642c
I20260926 19:39:40.422127 8126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6: Bootstrap starting.
I20260926 19:39:40.424942 8126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:40.425943 8126 log.cc:824] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:40.428306 8126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6: No bootstrap required, opened a new log
I20260926 19:39:40.431363 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:40.421032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39332 (local address 127.2.164.126:44301)
0926 19:39:40.421910 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:40.421920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:40.421960 (+ 40us) server_negotiation.cc:407] Connection header received
0926 19:39:40.423529 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:40.423553 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:40.423990 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:40.424312 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:40.424811 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:40.426172 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:40.427383 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:40.428037 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:40.428196 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:40.429296 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:40.429303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:40.429315 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:40.429795 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:40.429884 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:40.429907 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:40.430016 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:40.430217 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 19:39:40.430480 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":284,"threads_started":1}
I20260926 19:39:40.434979 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:39:40.436136 8126 raft_consensus.cc:348] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09e803e92384c849eecb2bbef6872e6" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44301 } }
I20260926 19:39:40.436434 8126 raft_consensus.cc:374] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:40.436520 8126 raft_consensus.cc:729] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09e803e92384c849eecb2bbef6872e6, State: Initialized, Role: FOLLOWER
I20260926 19:39:40.437062 8126 consensus_queue.cc:260] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09e803e92384c849eecb2bbef6872e6" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44301 } }
I20260926 19:39:40.437340 8126 raft_consensus.cc:388] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:40.437464 8126 raft_consensus.cc:482] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:40.437618 8126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:40.439597 8126 raft_consensus.cc:504] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09e803e92384c849eecb2bbef6872e6" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44301 } }
I20260926 19:39:40.440029 8126 leader_election.cc:302] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d09e803e92384c849eecb2bbef6872e6; no voters:
I20260926 19:39:40.440515 8126 leader_election.cc:288] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:40.440625 8131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:40.441253 8131 raft_consensus.cc:686] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [term 1 LEADER]: Becoming Leader. State: Replica: d09e803e92384c849eecb2bbef6872e6, State: Running, Role: LEADER
I20260926 19:39:40.441752 8131 consensus_queue.cc:237] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09e803e92384c849eecb2bbef6872e6" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44301 } }
I20260926 19:39:40.442307 8126 sys_catalog.cc:564] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:40.444024 8132 sys_catalog.cc:455] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d09e803e92384c849eecb2bbef6872e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09e803e92384c849eecb2bbef6872e6" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44301 } } }
I20260926 19:39:40.444422 8132 sys_catalog.cc:458] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:40.444432 8133 sys_catalog.cc:455] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d09e803e92384c849eecb2bbef6872e6. Latest consensus state: current_term: 1 leader_uuid: "d09e803e92384c849eecb2bbef6872e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d09e803e92384c849eecb2bbef6872e6" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44301 } } }
I20260926 19:39:40.444975 8133 sys_catalog.cc:458] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:40.449405 8137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:40.454391 8137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:40.461918 8137 catalog_manager.cc:1347] Generated new cluster ID: 008573896ee54468af7f27028bed3e10
I20260926 19:39:40.462033 8137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:40.487948 8137 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:40.489691 8137 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:40.500823 8137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d09e803e92384c849eecb2bbef6872e6: Generated new TSK 0
I20260926 19:39:40.501838 8137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:39:40.523032 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:40.514910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.2.164.126:44301)
0926 19:39:40.515090 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:40.515097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:40.515394 (+ 297us) server_negotiation.cc:407] Connection header received
0926 19:39:40.515630 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:40.515634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:40.515704 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:40.515865 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0926 19:39:40.516284 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:40.516989 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:40.518325 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:40.518776 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:40.518894 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:40.519341 (+ 447us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 19:39:40.521395 (+ 2054us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 19:39:40.522708 (+ 1313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 19:39:40.523149 8129 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 19:39:40.524554 8155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:39:40.514752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44301 (local address 127.0.0.1:39348)
0926 19:39:40.515300 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:40.515316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:40.515449 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:40.515895 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:40.515900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:40.515912 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 19:39:40.516146 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:40.516154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:40.517126 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:40.517132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:40.518192 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:40.518204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:40.518952 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:40.518959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:40.519089 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:40.519121 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 19:39:40.521601 (+ 2480us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 19:39:40.524103 (+ 2502us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.164.126:44301: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46111/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":2195,"thread_start_us":340,"threads_started":1}
I20260926 19:39:40.525806 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 8060
I20260926 19:39:40.526834 8121 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 19:39:40.535517 8156 webserver.cc:533] Webserver: sslize failed:
E20260926 19:39:40.535609 8156 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 19:39:40.615509 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 8060
2026-09-26T19:39:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (665 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T19:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:40Z Disabled control of system clock
I20260926 19:39:40.688141 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38139
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:38139
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/testchainca.cert with env {}
W20260926 19:39:40.984639 8161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:40.984923 8161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:40.985045 8161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:40.992105 8161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:40.992206 8161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:40.992247 8161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:40.992324 8161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:40.992385 8161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 19:39:40.992465 8161 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 19:39:40.992511 8161 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 19:39:41.002130 8161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:38139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:41.003366 8161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:41.004571 8161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:41.015810 8169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:41.015882 8166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:41.015774 8167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:41.016448 8161 server_base.cc:1034] running on GCE node
I20260926 19:39:41.017128 8161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:41.018469 8161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:41.019660 8161 hybrid_clock.cc:648] HybridClock initialized: now 1790451581019614 us; error 34 us; skew 500 ppm
I20260926 19:39:41.019994 8161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:39:41.022076 8161 webserver.cc:460] Webserver started at http://127.2.164.126:33411/ using document root <none> and password file <none>
I20260926 19:39:41.022701 8161 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:41.022800 8161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:41.023080 8161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:41.025760 8161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "9b820bdc83c3420faac092a1e5af24dc"
format_stamp: "Formatted at 2026-09-26 19:39:41 on dist-test-slave-mrw4"
server_key: "46f80f8531c1c668f600f97d30c40b35"
server_key_iv: "f000946c5cbaf5b7a9498b351784dd11"
server_key_version: "encryptionkey@0"
I20260926 19:39:41.026484 8161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "9b820bdc83c3420faac092a1e5af24dc"
format_stamp: "Formatted at 2026-09-26 19:39:41 on dist-test-slave-mrw4"
server_key: "46f80f8531c1c668f600f97d30c40b35"
server_key_iv: "f000946c5cbaf5b7a9498b351784dd11"
server_key_version: "encryptionkey@0"
I20260926 19:39:41.031211 8161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 19:39:41.034581 8175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:41.035944 8161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:39:41.036101 8161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "9b820bdc83c3420faac092a1e5af24dc"
format_stamp: "Formatted at 2026-09-26 19:39:41 on dist-test-slave-mrw4"
server_key: "46f80f8531c1c668f600f97d30c40b35"
server_key_iv: "f000946c5cbaf5b7a9498b351784dd11"
server_key_version: "encryptionkey@0"
I20260926 19:39:41.036301 8161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:41.058334 8161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:41.059170 8161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:41.059422 8161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:41.076135 8161 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:38139
I20260926 19:39:41.076143 8226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:38139 every 8 connection(s)
I20260926 19:39:41.077428 8161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:41.081022 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:41.082301 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 8161
I20260926 19:39:41.082450 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:41.083042 2705 external_mini_cluster.cc:1428] Setting key 6cd225af1bebec42dc2ad3571aee211f
I20260926 19:39:41.092167 8227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc: Bootstrap starting.
I20260926 19:39:41.095453 8227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:41.095889 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:41.084867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.2.164.126:38139)
0926 19:39:41.085836 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:41.085847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:41.085888 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:39:41.087438 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:41.087458 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:41.087913 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:41.088211 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:41.088767 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:41.090119 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:41.091807 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:41.092476 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:41.092595 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:41.093757 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:41.093765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:41.093801 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:39:41.094290 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:41.094370 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:39:41.094389 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:41.094473 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:41.094665 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 19:39:41.094974 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":354,"threads_started":1}
I20260926 19:39:41.096593 8227 log.cc:824] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:41.099558 8227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc: No bootstrap required, opened a new log
I20260926 19:39:41.099645 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:39:41.106433 8227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b820bdc83c3420faac092a1e5af24dc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38139 } }
I20260926 19:39:41.106778 8227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:41.106881 8227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b820bdc83c3420faac092a1e5af24dc, State: Initialized, Role: FOLLOWER
I20260926 19:39:41.107409 8227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b820bdc83c3420faac092a1e5af24dc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38139 } }
I20260926 19:39:41.107604 8227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:41.107731 8227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:41.107882 8227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:41.109846 8227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b820bdc83c3420faac092a1e5af24dc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38139 } }
I20260926 19:39:41.110283 8227 leader_election.cc:302] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b820bdc83c3420faac092a1e5af24dc; no voters:
I20260926 19:39:41.110810 8227 leader_election.cc:288] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:41.110917 8232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:41.111944 8232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [term 1 LEADER]: Becoming Leader. State: Replica: 9b820bdc83c3420faac092a1e5af24dc, State: Running, Role: LEADER
I20260926 19:39:41.112401 8232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b820bdc83c3420faac092a1e5af24dc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38139 } }
I20260926 19:39:41.112622 8227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:41.114853 8234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b820bdc83c3420faac092a1e5af24dc. Latest consensus state: current_term: 1 leader_uuid: "9b820bdc83c3420faac092a1e5af24dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b820bdc83c3420faac092a1e5af24dc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38139 } } }
I20260926 19:39:41.115190 8234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:41.115622 8233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b820bdc83c3420faac092a1e5af24dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b820bdc83c3420faac092a1e5af24dc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38139 } } }
I20260926 19:39:41.115952 8233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:41.116031 8239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:41.121567 8239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:41.130664 8239 catalog_manager.cc:1347] Generated new cluster ID: 2286c0a0e1554c29945512b1fad79470
I20260926 19:39:41.130790 8239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:41.141386 8239 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:41.142571 8239 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:41.153796 8239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b820bdc83c3420faac092a1e5af24dc: Generated new TSK 0
I20260926 19:39:41.154562 8239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:39:41.248313 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:41.168863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.2.164.126:38139)
0926 19:39:41.169004 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:41.169009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:41.169337 (+ 328us) server_negotiation.cc:407] Connection header received
0926 19:39:41.169544 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:41.169549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:41.169637 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:41.169870 (+ 233us) server_negotiation.cc:226] Negotiated authn=JWT
0926 19:39:41.170254 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:41.171383 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:41.172708 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:41.173416 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:41.173537 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:41.173954 (+ 417us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 19:39:41.247600 (+ 73646us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 19:39:41.248014 (+ 414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34230: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 19:39:41.248445 8230 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34230: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 19:39:41.248468 8256 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:39:41.168721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:38139 (local address 127.0.0.1:34230)
0926 19:39:41.169249 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:41.169264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:41.169384 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:41.169859 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:41.169863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:41.169871 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 19:39:41.170124 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:41.170132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:41.171493 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:41.171497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:41.172570 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:41.172582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:41.173553 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:41.173557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:41.173666 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:41.173699 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 19:39:41.247907 (+ 74208us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 19:39:41.248138 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.164.126:38139: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":264,"threads_started":1}
I20260926 19:39:41.249481 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 8161
I20260926 19:39:41.250602 8222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:41.345641 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 8161
2026-09-26T19:39:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (709 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (1350 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (376 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:43 dist-test-slave-mrw4 krb5kdc[8272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:43 dist-test-slave-mrw4 krb5kdc[8272](info): set up 2 sockets
Sep 26 19:39:43 dist-test-slave-mrw4 krb5kdc[8272](info): commencing operation
krb5kdc: starting...
W20260926 19:39:44.065529 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:44 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:39:44.248497 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35231
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:35231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:44.548835 8285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:44.549114 8285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:44.549253 8285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:44.556368 8285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:44.556478 8285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:44.556561 8285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:44.556638 8285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:44.556695 8285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:44.566128 8285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:35231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:44.567315 8285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:44.568521 8285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:44.579203 8290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:44.579257 8293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:44.579278 8291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:44.579875 8285 server_base.cc:1034] running on GCE node
I20260926 19:39:44.580562 8285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:44.581866 8285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:44.583048 8285 hybrid_clock.cc:648] HybridClock initialized: now 1790451584582993 us; error 46 us; skew 500 ppm
I20260926 19:39:44.583344 8285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:44 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451584, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:44.608623 8285 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:39:44.610054 8285 webserver.cc:460] Webserver started at http://127.2.164.126:40129/ using document root <none> and password file <none>
I20260926 19:39:44.610688 8285 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:44.610780 8285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:44.611025 8285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:44.613819 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "9571ff0b675e4c37b902e0ec29325661"
format_stamp: "Formatted at 2026-09-26 19:39:44 on dist-test-slave-mrw4"
server_key: "5ff9673199e7d2402117482104d1ac2c"
server_key_iv: "4271882b8d617cc66b1937cfe99e54df"
server_key_version: "encryptionkey@0"
I20260926 19:39:44.614490 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "9571ff0b675e4c37b902e0ec29325661"
format_stamp: "Formatted at 2026-09-26 19:39:44 on dist-test-slave-mrw4"
server_key: "5ff9673199e7d2402117482104d1ac2c"
server_key_iv: "4271882b8d617cc66b1937cfe99e54df"
server_key_version: "encryptionkey@0"
I20260926 19:39:44.619148 8285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 19:39:44.622175 8300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:44.623374 8285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:39:44.623518 8285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "9571ff0b675e4c37b902e0ec29325661"
format_stamp: "Formatted at 2026-09-26 19:39:44 on dist-test-slave-mrw4"
server_key: "5ff9673199e7d2402117482104d1ac2c"
server_key_iv: "4271882b8d617cc66b1937cfe99e54df"
server_key_version: "encryptionkey@0"
I20260926 19:39:44.623667 8285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:44.649550 8285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:44.653076 8285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:44.653332 8285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:44.669518 8285 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:35231
I20260926 19:39:44.669549 8351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:35231 every 8 connection(s)
I20260926 19:39:44.670703 8285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:44.673684 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 8285
I20260926 19:39:44.673851 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:44.674080 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:44.674464 2705 external_mini_cluster.cc:1428] Setting key 75d34d1bb3cdf86a0b3d620b2efb8606
I20260926 19:39:44.686980 8352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661: Bootstrap starting.
Sep 26 19:39:44 dist-test-slave-mrw4 krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:44.690299 8352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:44.691493 8352 log.cc:824] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:44.694485 8352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661: No bootstrap required, opened a new log
I20260926 19:39:44.695848 8355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:44.676398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.2.164.126:35231)
0926 19:39:44.677123 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:44.677136 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:44.677202 (+ 66us) server_negotiation.cc:407] Connection header received
0926 19:39:44.678708 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:44.678733 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:44.679201 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:44.679599 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:44.680692 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:44.682576 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:44.683725 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:44.684737 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:44.684914 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:44.688318 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:44.688328 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:44.688344 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:44.688392 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:44.691834 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:44.692585 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:44.692590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:44.692595 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:44.692770 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:44.693105 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:44.693110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:44.693113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:44.693739 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:44.694012 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:44.694293 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 19:39:44.694659 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":299,"threads_started":1}
I20260926 19:39:44.702440 8352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9571ff0b675e4c37b902e0ec29325661" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35231 } }
I20260926 19:39:44.702742 8352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:44.702809 8352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9571ff0b675e4c37b902e0ec29325661, State: Initialized, Role: FOLLOWER
I20260926 19:39:44.703379 8352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9571ff0b675e4c37b902e0ec29325661" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35231 } }
I20260926 19:39:44.703591 8352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:44.703686 8352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:44.703800 8352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:44.705931 8352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9571ff0b675e4c37b902e0ec29325661" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35231 } }
I20260926 19:39:44.706395 8352 leader_election.cc:302] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9571ff0b675e4c37b902e0ec29325661; no voters:
I20260926 19:39:44.706884 8352 leader_election.cc:288] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:44.707003 8357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:44.707571 8357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [term 1 LEADER]: Becoming Leader. State: Replica: 9571ff0b675e4c37b902e0ec29325661, State: Running, Role: LEADER
I20260926 19:39:44.708073 8357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9571ff0b675e4c37b902e0ec29325661" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35231 } }
I20260926 19:39:44.708709 8352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:44.710407 8358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9571ff0b675e4c37b902e0ec29325661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9571ff0b675e4c37b902e0ec29325661" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35231 } } }
I20260926 19:39:44.711365 8358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:44.711117 8359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9571ff0b675e4c37b902e0ec29325661. Latest consensus state: current_term: 1 leader_uuid: "9571ff0b675e4c37b902e0ec29325661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9571ff0b675e4c37b902e0ec29325661" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35231 } } }
I20260926 19:39:44.711504 8359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:44.712155 8365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:44.717600 8365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:44.725389 8365 catalog_manager.cc:1347] Generated new cluster ID: 551f5f08cda8482ca49ea2fef1a43e07
I20260926 19:39:44.725505 8365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:44.762738 8365 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:44.764524 8365 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:44.776701 8365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9571ff0b675e4c37b902e0ec29325661: Generated new TSK 0
I20260926 19:39:44.777518 8365 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:39:44.825933 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35231
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:45.129220 8378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:45.129469 8378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:45.129602 8378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:45.136430 8378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:45.136529 8378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:45.136673 8378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:45.145879 8378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:45.147116 8378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:45.148402 8378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:45.160355 8386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:45.160434 8384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:45.160410 8383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:45.161084 8378 server_base.cc:1034] running on GCE node
I20260926 19:39:45.161643 8378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:45.162474 8378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:45.163663 8378 hybrid_clock.cc:648] HybridClock initialized: now 1790451585163623 us; error 48 us; skew 500 ppm
I20260926 19:39:45.163944 8378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:45 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451585, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:45.166972 8378 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:39:45.168283 8378 webserver.cc:460] Webserver started at http://127.2.164.65:39845/ using document root <none> and password file <none>
I20260926 19:39:45.168861 8378 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:45.168962 8378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:45.169236 8378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:45.171866 8378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "ca9a50f7bfbb48ae8bc1687110316c63"
format_stamp: "Formatted at 2026-09-26 19:39:45 on dist-test-slave-mrw4"
server_key: "aaf6ee3fe6264458237b9a6c08cf4dbe"
server_key_iv: "136cb53f195fbd3ef6f4f59c740bd647"
server_key_version: "encryptionkey@0"
I20260926 19:39:45.172545 8378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "ca9a50f7bfbb48ae8bc1687110316c63"
format_stamp: "Formatted at 2026-09-26 19:39:45 on dist-test-slave-mrw4"
server_key: "aaf6ee3fe6264458237b9a6c08cf4dbe"
server_key_iv: "136cb53f195fbd3ef6f4f59c740bd647"
server_key_version: "encryptionkey@0"
I20260926 19:39:45.177273 8378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 19:39:45.180495 8393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:45.181824 8378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:39:45.181991 8378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "ca9a50f7bfbb48ae8bc1687110316c63"
format_stamp: "Formatted at 2026-09-26 19:39:45 on dist-test-slave-mrw4"
server_key: "aaf6ee3fe6264458237b9a6c08cf4dbe"
server_key_iv: "136cb53f195fbd3ef6f4f59c740bd647"
server_key_version: "encryptionkey@0"
I20260926 19:39:45.182171 8378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:45.202167 8378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:45.205948 8378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:45.206174 8378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:45.206997 8378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:45.208175 8378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:45.208242 8378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:45.208328 8378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:45.208369 8378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:45.236562 8378 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:42651
I20260926 19:39:45.236609 8505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:42651 every 8 connection(s)
I20260926 19:39:45.237742 8378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:45.243448 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 8378
I20260926 19:39:45.243638 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:45.244230 2705 external_mini_cluster.cc:1428] Setting key 80dcc415cc0c6e720951b04622e56794
Sep 26 19:39:45 dist-test-slave-mrw4 krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451585, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:45.256317 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:45.240116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:48309 (local address 127.2.164.126:35231)
0926 19:39:45.240533 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:45.240537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:45.241503 (+ 966us) server_negotiation.cc:407] Connection header received
0926 19:39:45.243241 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:45.243247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:45.243329 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:45.243454 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:45.245043 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:45.245891 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:45.247395 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:45.247773 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:45.247895 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:45.251261 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:45.251265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:45.251269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:45.251299 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:45.253659 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:45.254287 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:45.254292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:45.254296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:45.254400 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:45.254770 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:45.254775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:45.254778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:45.255029 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:45.255152 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:45.255762 (+ 610us) server_negotiation.cc:299] Negotiation successful
0926 19:39:45.255960 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":210,"threads_started":1}
I20260926 19:39:45.257225 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:45.240637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35231 (local address 127.2.164.65:48309)
0926 19:39:45.241353 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:45.241401 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:45.242975 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:45.243652 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:45.243660 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:45.244191 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:45.244871 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:45.244898 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:45.246042 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:45.246047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:45.247251 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:45.247263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:45.247988 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:45.247996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:45.248212 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:45.248848 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:45.248876 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:45.251061 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:45.253783 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:45.253791 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:45.253807 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:45.254182 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:45.254501 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:45.254506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:45.254510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:45.254673 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:45.255144 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:45.255151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:45.255440 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 19:39:45.255671 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 19:39:45.256124 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":257,"threads_started":1}
I20260926 19:39:45.258701 8506 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35231
I20260926 19:39:45.259029 8506 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:45.259894 8506 heartbeater.cc:511] Master 127.2.164.126:35231 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260926 19:39:45.262010 8317 ts_manager.cc:194] Registered new tserver with Master: ca9a50f7bfbb48ae8bc1687110316c63 (127.2.164.65:42651)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260926 19:39:45.263536 8317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:48309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:39:45.283556 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35231
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:45.596341 8512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:45.596619 8512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:45.596741 8512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:45.603793 8512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:45.603883 8512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:45.604041 8512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:45.613471 8512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:45.614601 8512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:45.615767 8512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:45.627526 8520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:45.627564 8517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:45.627491 8518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:45.628511 8512 server_base.cc:1034] running on GCE node
I20260926 19:39:45.629004 8512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:45.629839 8512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:45.631055 8512 hybrid_clock.cc:648] HybridClock initialized: now 1790451585630999 us; error 48 us; skew 500 ppm
I20260926 19:39:45.631331 8512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:45 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451585, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:45.634100 8512 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:39:45.635365 8512 webserver.cc:460] Webserver started at http://127.2.164.66:43665/ using document root <none> and password file <none>
I20260926 19:39:45.635957 8512 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:45.636065 8512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:45.636324 8512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:45.638981 8512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "3e1b59d0acda40839d18f87ac1cef0e5"
format_stamp: "Formatted at 2026-09-26 19:39:45 on dist-test-slave-mrw4"
server_key: "2286eb55cf30f64fd022b273ce34ecea"
server_key_iv: "fa81b5e9ce3d07ec49c63e09f72179e9"
server_key_version: "encryptionkey@0"
I20260926 19:39:45.639690 8512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "3e1b59d0acda40839d18f87ac1cef0e5"
format_stamp: "Formatted at 2026-09-26 19:39:45 on dist-test-slave-mrw4"
server_key: "2286eb55cf30f64fd022b273ce34ecea"
server_key_iv: "fa81b5e9ce3d07ec49c63e09f72179e9"
server_key_version: "encryptionkey@0"
I20260926 19:39:45.644220 8512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 19:39:45.647264 8527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:45.648464 8512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 19:39:45.648598 8512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "3e1b59d0acda40839d18f87ac1cef0e5"
format_stamp: "Formatted at 2026-09-26 19:39:45 on dist-test-slave-mrw4"
server_key: "2286eb55cf30f64fd022b273ce34ecea"
server_key_iv: "fa81b5e9ce3d07ec49c63e09f72179e9"
server_key_version: "encryptionkey@0"
I20260926 19:39:45.648761 8512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:45.680904 8512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:45.684484 8512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:45.684721 8512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:45.685699 8512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:45.686988 8512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:45.687060 8512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:45.687175 8512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:45.687222 8512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:45.715483 8512 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:41167
I20260926 19:39:45.715517 8639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:41167 every 8 connection(s)
I20260926 19:39:45.716702 8512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:45.718608 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 8512
I20260926 19:39:45.718747 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:45.719213 2705 external_mini_cluster.cc:1428] Setting key 08acc17fe51adc65fa089859e41ec6c0
Sep 26 19:39:45 dist-test-slave-mrw4 krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451585, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:45.737185 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:45.719088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:38961 (local address 127.2.164.126:35231)
0926 19:39:45.719281 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:45.719287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:45.720590 (+ 1303us) server_negotiation.cc:407] Connection header received
0926 19:39:45.722849 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:45.722853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:45.722917 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:45.722995 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:45.724738 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:45.725514 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:45.727565 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:45.728190 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:45.728307 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:45.731491 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:45.731497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:45.731501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:45.731537 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:45.734421 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:45.735026 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:45.735031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:45.735035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:45.735181 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:45.735552 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:45.735556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:45.735559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:45.735898 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:45.736057 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:45.736624 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 19:39:45.736837 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:39:45.737890 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:45.719680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35231 (local address 127.2.164.66:38961)
0926 19:39:45.720427 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:45.720471 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:45.722572 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:45.723247 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:45.723256 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:45.723888 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:45.724586 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:45.724608 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:45.725653 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:45.725658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:45.727438 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:45.727452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:45.728378 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:45.728383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:45.728577 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:45.729186 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:45.729221 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:45.731289 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:45.734536 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:45.734542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:45.734553 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:45.734917 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:45.735278 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:45.735282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:45.735284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:45.735457 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:45.736004 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:45.736009 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:45.736280 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 19:39:45.736499 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 19:39:45.736843 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":324,"threads_started":1}
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260926 19:39:45.739110 8640 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35231
I20260926 19:39:45.739370 8640 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:45.740022 8640 heartbeater.cc:511] Master 127.2.164.126:35231 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
I20260926 19:39:45.741308 8317 ts_manager.cc:194] Registered new tserver with Master: 3e1b59d0acda40839d18f87ac1cef0e5 (127.2.164.66:41167)
I20260926 19:39:45.742249 8317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:38961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:39:45.762122 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35231
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:46.062268 8645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:46.062539 8645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:46.062673 8645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:46.069746 8645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:46.069859 8645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:46.070001 8645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:46.079600 8645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:46.080812 8645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:46.082093 8645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:46.094077 8650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:46.094049 8653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:46.094120 8651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:46.094758 8645 server_base.cc:1034] running on GCE node
I20260926 19:39:46.095259 8645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:46.096062 8645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:46.097259 8645 hybrid_clock.cc:648] HybridClock initialized: now 1790451586097210 us; error 42 us; skew 500 ppm
I20260926 19:39:46.097534 8645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:46.100302 8645 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:39:46.101676 8645 webserver.cc:460] Webserver started at http://127.2.164.67:46085/ using document root <none> and password file <none>
I20260926 19:39:46.102275 8645 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:46.102370 8645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:46.102612 8645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:46.105289 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "ddee65aaa58d42d3a7ce37467e89a4fc"
format_stamp: "Formatted at 2026-09-26 19:39:46 on dist-test-slave-mrw4"
server_key: "524f62f2f74106337b10cb5416e6dec6"
server_key_iv: "3c19a372d045c4bfed594785e69a9b82"
server_key_version: "encryptionkey@0"
I20260926 19:39:46.106022 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "ddee65aaa58d42d3a7ce37467e89a4fc"
format_stamp: "Formatted at 2026-09-26 19:39:46 on dist-test-slave-mrw4"
server_key: "524f62f2f74106337b10cb5416e6dec6"
server_key_iv: "3c19a372d045c4bfed594785e69a9b82"
server_key_version: "encryptionkey@0"
I20260926 19:39:46.110671 8645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:39:46.113845 8660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:46.115088 8645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:39:46.115221 8645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "ddee65aaa58d42d3a7ce37467e89a4fc"
format_stamp: "Formatted at 2026-09-26 19:39:46 on dist-test-slave-mrw4"
server_key: "524f62f2f74106337b10cb5416e6dec6"
server_key_iv: "3c19a372d045c4bfed594785e69a9b82"
server_key_version: "encryptionkey@0"
I20260926 19:39:46.115383 8645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:46.151230 8645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:46.155073 8645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:46.155320 8645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:46.156200 8645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:46.157480 8645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:46.157554 8645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:46.157667 8645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:46.157716 8645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:46.185724 8645 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:35791
I20260926 19:39:46.185786 8772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:35791 every 8 connection(s)
I20260926 19:39:46.186951 8645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:46.187265 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 8645
I20260926 19:39:46.187439 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:46.187987 2705 external_mini_cluster.cc:1428] Setting key 786548d8dd6b2c19513ae17e3cccf4ec
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:39:46.205652 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:46.189653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:57641 (local address 127.2.164.126:35231)
0926 19:39:46.189842 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:46.189848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:46.191070 (+ 1222us) server_negotiation.cc:407] Connection header received
0926 19:39:46.192727 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:46.192731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:46.192798 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:46.192879 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:46.194253 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.194960 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.196448 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.196839 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.196929 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:46.199987 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:46.199992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:46.199995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:46.200026 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:46.202683 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.203458 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.203464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.203468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.203619 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.204003 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.204008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.204011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.204284 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:46.204427 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:46.205059 (+ 632us) server_negotiation.cc:299] Negotiation successful
0926 19:39:46.205312 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:39:46.206718 8775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:46.190216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35231 (local address 127.2.164.67:57641)
0926 19:39:46.190951 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:46.190987 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:46.192499 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:46.193042 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:46.193049 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:46.193577 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:46.194116 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:46.194136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.195112 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.195116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:46.196318 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:46.196331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.196998 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.197003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:46.197238 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:46.197851 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:46.197872 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:46.199807 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:46.202847 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:46.202856 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:46.202872 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:46.203357 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:46.203717 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:46.203722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:46.203725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:46.203892 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:46.204403 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:46.204410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:46.204731 (+ 321us) client_negotiation.cc:769] Sending connection context
0926 19:39:46.204964 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 19:39:46.205504 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":294,"threads_started":1}
I20260926 19:39:46.207945 8773 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35231
I20260926 19:39:46.208217 8773 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:46.208904 8773 heartbeater.cc:511] Master 127.2.164.126:35231 requested a full tablet report, sending...
I20260926 19:39:46.210253 8317 ts_manager.cc:194] Registered new tserver with Master: ddee65aaa58d42d3a7ce37467e89a4fc (127.2.164.67:35791)
I20260926 19:39:46.211042 8317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:57641
I20260926 19:39:46.215402 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/test-admin.keytab.
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:46.238323 2705 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 19:39:46.243062 8780 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:39:46.241065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.2.164.65:42651)
0926 19:39:46.241626 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:46.241633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:46.241679 (+ 46us) server_negotiation.cc:407] Connection header received
0926 19:39:46.241733 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:46.241739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:46.241916 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:46.242030 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:46.242766 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34802: BlockingRecv error: recv got EOF from 127.0.0.1:34802 (error 108)
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":307,"threads_started":1}
W20260926 19:39:46.243284 2705 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/krb5cc' of type FILE
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:46.244669 2705 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:46.245905 2705 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:46.259719 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:46.247404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.2.164.65:42651)
0926 19:39:46.247574 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:46.247579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:46.247716 (+ 137us) server_negotiation.cc:407] Connection header received
0926 19:39:46.247862 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:46.247866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:46.247917 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:46.247985 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:46.248957 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.249889 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.251029 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.251541 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.251667 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:46.254980 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:46.254991 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:46.255004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:46.255039 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:46.257523 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.258103 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.258110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.258112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.258226 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.258516 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.258519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.258521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.258840 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:46.259065 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:46.259283 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 19:39:46.259470 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 19:39:46.265043 8780 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:39:46.263367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.2.164.65:42651)
0926 19:39:46.263527 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:46.263532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:46.263832 (+ 300us) server_negotiation.cc:407] Connection header received
0926 19:39:46.263988 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:46.263992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:46.264046 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:46.264120 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:46.264830 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34818: BlockingRecv error: recv got EOF from 127.0.0.1:34818 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 19:39:46.265208 2705 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/krb5cc' of type FILE
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:46.266292 8506 heartbeater.cc:503] Master 127.2.164.126:35231 was elected leader, sending a full tablet report...
I20260926 19:39:46.266683 2705 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:46.268003 2705 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:46.280491 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:46.269075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34824 (local address 127.2.164.65:42651)
0926 19:39:46.269238 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:46.269243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:46.269481 (+ 238us) server_negotiation.cc:407] Connection header received
0926 19:39:46.269622 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:46.269625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:46.269674 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:46.269746 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:46.270589 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.271241 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.272417 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.272842 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.272945 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:46.276152 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:46.276156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:46.276160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:46.276188 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:46.278401 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.278978 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.278983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.278985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.279103 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.279485 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.279489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.279491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.279737 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:46.279916 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:46.280123 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 19:39:46.280267 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 19:39:46.284161 8780 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:39:46.282648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.2.164.65:42651)
0926 19:39:46.282800 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:46.282806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:46.283051 (+ 245us) server_negotiation.cc:407] Connection header received
0926 19:39:46.283195 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:46.283198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:46.283262 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:46.283336 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:46.283984 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34840: BlockingRecv error: recv got EOF from 127.0.0.1:34840 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 19:39:46.284333 2705 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790451540687502-2705-0/krb5kdc/krb5cc' of type FILE
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:46.285861 2705 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:46.287035 2705 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:39:46.300000 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:46.288187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.2.164.65:42651)
0926 19:39:46.288374 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:46.288380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:46.288614 (+ 234us) server_negotiation.cc:407] Connection header received
0926 19:39:46.288759 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:46.288763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:46.288814 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:46.288885 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:46.289847 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.290517 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.291696 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.292121 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.292224 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:46.295455 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:46.295460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:46.295463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:46.295492 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:46.297728 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.298291 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.298296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.298299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.298419 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.298739 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.298744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.298747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.299019 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:46.299283 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:46.299540 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 19:39:46.299724 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:39:46.301366 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 8378
I20260926 19:39:46.311933 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:46.302005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.2.164.65:42651)
0926 19:39:46.302141 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:46.302147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:46.302419 (+ 272us) server_negotiation.cc:407] Connection header received
0926 19:39:46.302563 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:46.302568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:46.302617 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:46.302685 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:46.303532 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.304155 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.305351 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.305720 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.305828 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:46.307616 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:46.307621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:46.307623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:46.307650 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:46.309772 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.310312 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.310317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.310319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.310433 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.310759 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.310764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.310767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.310988 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:46.311215 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:46.311492 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 19:39:46.311663 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 19:39:46.312397 8501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:46.392637 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 8378
I20260926 19:39:46.405301 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 8512
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:39:46.419533 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:46.406074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.2.164.66:41167)
0926 19:39:46.406573 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:46.406580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:46.406607 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:39:46.406679 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:46.406685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:46.406894 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:46.407018 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:46.407990 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.409040 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.410388 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.410929 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.411025 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:46.414564 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:46.414575 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:46.414589 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:46.414628 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:46.417225 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.417801 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.417807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.417809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.417919 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.418228 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.418231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.418233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.418555 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:46.418785 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:46.419031 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 19:39:46.419218 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":229,"threads_started":1}
I20260926 19:39:46.420709 8635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:46.499226 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 8512
I20260926 19:39:46.511673 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 8645
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:39:46.526218 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:46.512463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.2.164.67:35791)
0926 19:39:46.513030 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:46.513037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:46.513064 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:39:46.513125 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:46.513131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:46.513366 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:46.513542 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:46.514546 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.515542 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.516773 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:46.517464 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:46.517585 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:46.521331 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:46.521341 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:46.521355 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:46.521387 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:46.523759 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.524370 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.524377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.524379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.524511 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:46.524841 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:46.524845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:46.524847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:46.525152 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:46.525456 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:46.525716 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 19:39:46.525946 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":254,"threads_started":1}
I20260926 19:39:46.527470 8768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:46.607432 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 8645
I20260926 19:39:46.619766 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 8285
I20260926 19:39:46.620817 8347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:46.720351 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 8285
2026-09-26T19:39:46Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3653 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8800](info): set up 2 sockets
Sep 26 19:39:46 dist-test-slave-mrw4 krb5kdc[8800](info): commencing operation
krb5kdc: starting...
W20260926 19:39:47.713511 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:47 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.126.keytab.
Entry for principal oryx/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.126.keytab.
I20260926 19:39:47.888326 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:39835
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.126.keytab
--principal=oryx/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:39835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:48.182418 8813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:48.182708 8813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:48.182840 8813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:48.189842 8813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:48.189963 8813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:48.190039 8813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:48.190114 8813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:48.190174 8813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:48.199584 8813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:39835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.126.keytab
--principal=oryx/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:39835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:48.200944 8813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:48.202207 8813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:48.212855 8819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:48.213001 8818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:48.212926 8821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:48.213902 8813 server_base.cc:1034] running on GCE node
I20260926 19:39:48.214600 8813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:48.215864 8813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:48.217042 8813 hybrid_clock.cc:648] HybridClock initialized: now 1790451588216991 us; error 42 us; skew 500 ppm
I20260926 19:39:48.217386 8813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:48 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451588, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:48.220208 8813 init.cc:377] Logged in from keytab as oryx/127.2.164.126@KRBTEST.COM (short username oryx)
I20260926 19:39:48.221562 8813 webserver.cc:460] Webserver started at http://127.2.164.126:41879/ using document root <none> and password file <none>
I20260926 19:39:48.222188 8813 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:48.222301 8813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:48.222568 8813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:48.225363 8813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "1547d4ec04a04ec58ebfae894e02340f"
format_stamp: "Formatted at 2026-09-26 19:39:48 on dist-test-slave-mrw4"
server_key: "41ba9a2b985476f23c1f26697891ccf4"
server_key_iv: "d3e15403343589d84eea2cdfea7cb68c"
server_key_version: "encryptionkey@0"
I20260926 19:39:48.226071 8813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "1547d4ec04a04ec58ebfae894e02340f"
format_stamp: "Formatted at 2026-09-26 19:39:48 on dist-test-slave-mrw4"
server_key: "41ba9a2b985476f23c1f26697891ccf4"
server_key_iv: "d3e15403343589d84eea2cdfea7cb68c"
server_key_version: "encryptionkey@0"
I20260926 19:39:48.230784 8813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:39:48.233916 8828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:48.235128 8813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:39:48.235282 8813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "1547d4ec04a04ec58ebfae894e02340f"
format_stamp: "Formatted at 2026-09-26 19:39:48 on dist-test-slave-mrw4"
server_key: "41ba9a2b985476f23c1f26697891ccf4"
server_key_iv: "d3e15403343589d84eea2cdfea7cb68c"
server_key_version: "encryptionkey@0"
I20260926 19:39:48.235440 8813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:48.264652 8813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:48.268637 8813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:48.268885 8813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:48.284849 8813 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:39835
I20260926 19:39:48.284878 8879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:39835 every 8 connection(s)
I20260926 19:39:48.286083 8813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:48.289556 8880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:48.292762 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 8813
I20260926 19:39:48.292924 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:48.293584 2705 external_mini_cluster.cc:1428] Setting key 6b90b001b27e5cd816350c4352bbe6de
I20260926 19:39:48.300582 8880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f: Bootstrap starting.
I20260926 19:39:48.303874 8880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:48.305037 8880 log.cc:824] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:48.307578 8880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f: No bootstrap required, opened a new log
Sep 26 19:39:48 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:39:48.316632 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:48.295423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.2.164.126:39835)
0926 19:39:48.296151 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:48.296165 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:48.296211 (+ 46us) server_negotiation.cc:407] Connection header received
0926 19:39:48.298981 (+ 2770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:48.299009 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:48.299452 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:48.299853 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:48.301024 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:48.302917 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:48.304042 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:48.305003 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:48.305195 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:48.308825 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:48.308834 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:48.308851 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:48.308893 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:48.312633 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:48.313273 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:48.313285 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:48.313292 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:48.313492 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:48.313794 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:48.313799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:48.313802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:48.314437 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:48.314704 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:48.314983 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 19:39:48.315376 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":299,"threads_started":1}
I20260926 19:39:48.317247 8880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } }
I20260926 19:39:48.317541 8880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:48.317631 8880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1547d4ec04a04ec58ebfae894e02340f, State: Initialized, Role: FOLLOWER
I20260926 19:39:48.318151 8880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } }
I20260926 19:39:48.318359 8880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:48.318477 8880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:48.318640 8880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:48.320590 8880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } }
I20260926 19:39:48.321002 8880 leader_election.cc:302] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1547d4ec04a04ec58ebfae894e02340f; no voters:
I20260926 19:39:48.321647 8880 leader_election.cc:288] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:48.321787 8885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:48.322402 8885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 1 LEADER]: Becoming Leader. State: Replica: 1547d4ec04a04ec58ebfae894e02340f, State: Running, Role: LEADER
I20260926 19:39:48.322904 8885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } }
I20260926 19:39:48.323719 8880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:48.325606 8886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } }
I20260926 19:39:48.325578 8887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1547d4ec04a04ec58ebfae894e02340f. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } }
I20260926 19:39:48.325971 8886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:48.326013 8887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:48.330421 8890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:48.334756 8890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:48.342418 8890 catalog_manager.cc:1347] Generated new cluster ID: cf759cf0b7094eb98ac9c84b7965c53e
I20260926 19:39:48.342512 8890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:48.361651 8890 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:48.362777 8890 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:48.370297 8890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f: Generated new TSK 0
I20260926 19:39:48.370999 8890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab.
I20260926 19:39:48.418977 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39835
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab
--principal=oryx/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:48.723981 8906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:48.724255 8906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:48.724390 8906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:48.731329 8906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:48.731417 8906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:48.731529 8906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:48.740721 8906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab
--principal=oryx/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:48.741873 8906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:48.743099 8906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:48.755182 8914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:48.755249 8912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:48.755613 8911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:48.755919 8906 server_base.cc:1034] running on GCE node
I20260926 19:39:48.756551 8906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:48.757443 8906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:48.758633 8906 hybrid_clock.cc:648] HybridClock initialized: now 1790451588758587 us; error 34 us; skew 500 ppm
I20260926 19:39:48.758915 8906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:48 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451588, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:48.761795 8906 init.cc:377] Logged in from keytab as oryx/127.2.164.65@KRBTEST.COM (short username oryx)
I20260926 19:39:48.763072 8906 webserver.cc:460] Webserver started at http://127.2.164.65:36699/ using document root <none> and password file <none>
I20260926 19:39:48.763684 8906 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:48.763777 8906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:48.764007 8906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:48.766716 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "b82043831211489bbc3c7f33beb81c91"
format_stamp: "Formatted at 2026-09-26 19:39:48 on dist-test-slave-mrw4"
server_key: "fb20b0d00d0086849708c0f1f68de6e4"
server_key_iv: "785d3e596dc2ec8ece28dd0e2f43ab7d"
server_key_version: "encryptionkey@0"
I20260926 19:39:48.767390 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "b82043831211489bbc3c7f33beb81c91"
format_stamp: "Formatted at 2026-09-26 19:39:48 on dist-test-slave-mrw4"
server_key: "fb20b0d00d0086849708c0f1f68de6e4"
server_key_iv: "785d3e596dc2ec8ece28dd0e2f43ab7d"
server_key_version: "encryptionkey@0"
I20260926 19:39:48.772114 8906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:39:48.775305 8921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:48.776544 8906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:39:48.776687 8906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "b82043831211489bbc3c7f33beb81c91"
format_stamp: "Formatted at 2026-09-26 19:39:48 on dist-test-slave-mrw4"
server_key: "fb20b0d00d0086849708c0f1f68de6e4"
server_key_iv: "785d3e596dc2ec8ece28dd0e2f43ab7d"
server_key_version: "encryptionkey@0"
I20260926 19:39:48.776860 8906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:48.790490 8906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:48.794049 8906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:48.794298 8906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:48.795137 8906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:48.796340 8906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:48.796413 8906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:48.796530 8906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:48.796586 8906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:48.825379 8906 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:45559
I20260926 19:39:48.825424 9033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:45559 every 8 connection(s)
I20260926 19:39:48.826530 8906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:48.827122 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 8906
I20260926 19:39:48.827325 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:48.828138 2705 external_mini_cluster.cc:1428] Setting key d10a9afa272aacaebd22eadbdca7ccce
Sep 26 19:39:48 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451588, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:39:48.846676 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:48.828926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:33577 (local address 127.2.164.126:39835)
0926 19:39:48.829340 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:48.829345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:48.830612 (+ 1267us) server_negotiation.cc:407] Connection header received
0926 19:39:48.832910 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:48.832914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:48.832975 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:48.833067 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:48.834803 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:48.835483 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:48.837388 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:48.837774 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:48.837884 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:48.841131 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:48.841136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:48.841139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:48.841199 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:48.843461 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:48.844372 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:48.844377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:48.844379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:48.844515 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:48.844974 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:48.844977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:48.844979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:48.845279 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:48.845384 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:48.846191 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 19:39:48.846348 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":239,"threads_started":1}
WARNING: no policy specified for oryx/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260926 19:39:48.847931 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:48.829597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.65:33577)
0926 19:39:48.830435 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:48.830486 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:48.832638 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:48.833331 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:48.833342 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:48.833947 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:48.834647 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:48.834670 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:48.835625 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:48.835629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:48.837256 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:48.837267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:48.837976 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:48.837983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:48.838216 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:48.838828 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:48.838848 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:48.840930 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:48.843645 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:48.843654 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:48.843669 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:48.844232 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:48.844623 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:48.844628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:48.844632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:48.844877 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:48.845389 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:48.845397 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:48.845790 (+ 393us) client_negotiation.cc:769] Sending connection context
0926 19:39:48.846098 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 19:39:48.846604 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":294,"threads_started":1}
I20260926 19:39:48.849303 9034 heartbeater.cc:347] Connected to a master server at 127.2.164.126:39835
I20260926 19:39:48.849628 9034 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.66@KRBTEST.COM" created.
I20260926 19:39:48.850435 9034 heartbeater.cc:511] Master 127.2.164.126:39835 requested a full tablet report, sending...
I20260926 19:39:48.852710 8845 ts_manager.cc:194] Registered new tserver with Master: b82043831211489bbc3c7f33beb81c91 (127.2.164.65:45559)
I20260926 19:39:48.854437 8845 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:33577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab.
I20260926 19:39:48.872889 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39835
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab
--principal=oryx/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:49.173128 9040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:49.173424 9040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:49.173560 9040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:49.180505 9040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:49.180594 9040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:49.180728 9040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:49.190116 9040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab
--principal=oryx/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:49.191331 9040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:49.192607 9040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:49.205034 9045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:49.205112 9048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:49.205062 9046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:49.205826 9040 server_base.cc:1034] running on GCE node
I20260926 19:39:49.206421 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:49.207279 9040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:49.208482 9040 hybrid_clock.cc:648] HybridClock initialized: now 1790451589208420 us; error 54 us; skew 500 ppm
I20260926 19:39:49.208775 9040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:49 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451589, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:49.211796 9040 init.cc:377] Logged in from keytab as oryx/127.2.164.66@KRBTEST.COM (short username oryx)
I20260926 19:39:49.213076 9040 webserver.cc:460] Webserver started at http://127.2.164.66:36407/ using document root <none> and password file <none>
I20260926 19:39:49.213723 9040 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:49.213835 9040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:49.214107 9040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:49.216807 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "a3a543749a3e4f2fa93f5de7075a88d3"
format_stamp: "Formatted at 2026-09-26 19:39:49 on dist-test-slave-mrw4"
server_key: "721d8eb9885d55e4e42ce7b93ebeb7e8"
server_key_iv: "2423ab013ff041cff1be87725278f514"
server_key_version: "encryptionkey@0"
I20260926 19:39:49.217531 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "a3a543749a3e4f2fa93f5de7075a88d3"
format_stamp: "Formatted at 2026-09-26 19:39:49 on dist-test-slave-mrw4"
server_key: "721d8eb9885d55e4e42ce7b93ebeb7e8"
server_key_iv: "2423ab013ff041cff1be87725278f514"
server_key_version: "encryptionkey@0"
I20260926 19:39:49.222239 9040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:39:49.225540 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:49.226763 9040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 19:39:49.226897 9040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "a3a543749a3e4f2fa93f5de7075a88d3"
format_stamp: "Formatted at 2026-09-26 19:39:49 on dist-test-slave-mrw4"
server_key: "721d8eb9885d55e4e42ce7b93ebeb7e8"
server_key_iv: "2423ab013ff041cff1be87725278f514"
server_key_version: "encryptionkey@0"
I20260926 19:39:49.227065 9040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:49.238870 9040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:49.242377 9040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:49.242612 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:49.243420 9040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:49.244581 9040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:49.244654 9040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:49.244765 9040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:49.244817 9040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:49.272945 9040 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:34565
I20260926 19:39:49.272962 9167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:34565 every 8 connection(s)
I20260926 19:39:49.274142 9040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:49.278712 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 9040
I20260926 19:39:49.278863 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:49.279270 2705 external_mini_cluster.cc:1428] Setting key 5837a493a2777fcece06cd9314949dc2
Sep 26 19:39:49 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451589, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:39:49.293092 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:49.276449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:35169 (local address 127.2.164.126:39835)
0926 19:39:49.276605 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:49.276609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:49.277679 (+ 1070us) server_negotiation.cc:407] Connection header received
0926 19:39:49.279431 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:49.279436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:49.279521 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:49.279612 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:49.281332 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:49.282001 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:49.283481 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:49.283878 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:49.283980 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:49.287501 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:49.287507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:49.287513 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:49.287551 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:49.290307 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:49.290969 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:49.290973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:49.290975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:49.291113 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:49.291497 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:49.291500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:49.291503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:49.291763 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:49.291888 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:49.292533 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 19:39:49.292759 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:39:49.294202 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:49.276877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.66:35169)
0926 19:39:49.277539 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:49.277575 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:49.279182 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:49.279862 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:49.279870 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:49.280462 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:49.281139 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:49.281158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:49.282146 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:49.282149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:49.283344 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:49.283356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:49.284076 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:49.284083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:49.284303 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:49.284904 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:49.284926 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:49.287269 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:49.290431 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:49.290439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:49.290454 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:49.290849 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:49.291211 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:49.291216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:49.291220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:49.291388 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:49.291876 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:49.291884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:49.292194 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 19:39:49.292423 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 19:39:49.292920 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":284,"threads_started":1}
I20260926 19:39:49.295547 9168 heartbeater.cc:347] Connected to a master server at 127.2.164.126:39835
I20260926 19:39:49.295838 9168 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:49.296523 9168 heartbeater.cc:511] Master 127.2.164.126:39835 requested a full tablet report, sending...
I20260926 19:39:49.297880 8845 ts_manager.cc:194] Registered new tserver with Master: a3a543749a3e4f2fa93f5de7075a88d3 (127.2.164.66:34565)
I20260926 19:39:49.298681 8845 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:35169
WARNING: no policy specified for oryx/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab.
I20260926 19:39:49.324025 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39835
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab
--principal=oryx/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:49.630295 9173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:49.630563 9173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:49.630712 9173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:49.637619 9173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:49.637724 9173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:49.637873 9173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:49.647125 9173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab
--principal=oryx/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:39835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:49.648355 9173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:49.649586 9173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:49.661536 9179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:49.661583 9181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:49.661607 9178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:49.662279 9173 server_base.cc:1034] running on GCE node
I20260926 19:39:49.662796 9173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:49.663651 9173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:49.664839 9173 hybrid_clock.cc:648] HybridClock initialized: now 1790451589664798 us; error 35 us; skew 500 ppm
I20260926 19:39:49.665115 9173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:49 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451589, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:49.671623 9173 init.cc:377] Logged in from keytab as oryx/127.2.164.67@KRBTEST.COM (short username oryx)
I20260926 19:39:49.672981 9173 webserver.cc:460] Webserver started at http://127.2.164.67:36659/ using document root <none> and password file <none>
I20260926 19:39:49.673655 9173 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:49.673764 9173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:49.674026 9173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:49.676698 9173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "5cf6d073c00548fc93ebea3723743c47"
format_stamp: "Formatted at 2026-09-26 19:39:49 on dist-test-slave-mrw4"
server_key: "2c935f333aba468152d4bfa12504c055"
server_key_iv: "ba565ab3e36a5520e0fe8892c9f720e5"
server_key_version: "encryptionkey@0"
I20260926 19:39:49.677433 9173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "5cf6d073c00548fc93ebea3723743c47"
format_stamp: "Formatted at 2026-09-26 19:39:49 on dist-test-slave-mrw4"
server_key: "2c935f333aba468152d4bfa12504c055"
server_key_iv: "ba565ab3e36a5520e0fe8892c9f720e5"
server_key_version: "encryptionkey@0"
I20260926 19:39:49.682116 9173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:39:49.685149 9188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:49.686378 9173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 19:39:49.686513 9173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "5cf6d073c00548fc93ebea3723743c47"
format_stamp: "Formatted at 2026-09-26 19:39:49 on dist-test-slave-mrw4"
server_key: "2c935f333aba468152d4bfa12504c055"
server_key_iv: "ba565ab3e36a5520e0fe8892c9f720e5"
server_key_version: "encryptionkey@0"
I20260926 19:39:49.686677 9173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:49.705379 9173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:49.708956 9173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:49.709244 9173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:49.710112 9173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:49.711380 9173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:49.711452 9173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:49.711566 9173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:49.711614 9173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:49.740257 9173 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:42421
I20260926 19:39:49.740298 9300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:42421 every 8 connection(s)
I20260926 19:39:49.741487 9173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:49.742122 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 9173
I20260926 19:39:49.742290 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:49.742818 2705 external_mini_cluster.cc:1428] Setting key 06b9751910906cab78fe958b0f2eea7f
Sep 26 19:39:49 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451589, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:39:49.759600 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:49.743891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:59869 (local address 127.2.164.126:39835)
0926 19:39:49.744076 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:49.744082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:49.745435 (+ 1353us) server_negotiation.cc:407] Connection header received
0926 19:39:49.747177 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:49.747181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:49.747245 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:49.747321 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:49.748667 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:49.749503 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:49.750996 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:49.751398 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:49.751487 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:49.754434 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:49.754439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:49.754443 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:49.754473 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:49.757199 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:49.757913 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:49.757918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:49.757920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:49.758048 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:49.758379 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:49.758382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:49.758384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:49.758620 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:49.758719 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:49.759225 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 19:39:49.759365 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:39:49.760414 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:49.744509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.67:59869)
0926 19:39:49.745284 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:49.745333 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:49.746948 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:49.747505 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:49.747513 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:49.747999 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:49.748521 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:49.748538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:49.749675 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:49.749678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:49.750872 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:49.750883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:49.751566 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:49.751571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:49.751759 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:49.752281 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:49.752305 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:49.754253 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:49.757311 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:49.757318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:49.757331 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:49.757823 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:49.758148 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:49.758151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:49.758153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:49.758307 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:49.758703 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:49.758708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:49.758964 (+ 256us) client_negotiation.cc:769] Sending connection context
0926 19:39:49.759165 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 19:39:49.759477 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":292,"threads_started":1}
I20260926 19:39:49.761533 9301 heartbeater.cc:347] Connected to a master server at 127.2.164.126:39835
I20260926 19:39:49.761812 9301 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:49.762431 9301 heartbeater.cc:511] Master 127.2.164.126:39835 requested a full tablet report, sending...
I20260926 19:39:49.763649 8845 ts_manager.cc:194] Registered new tserver with Master: 5cf6d073c00548fc93ebea3723743c47 (127.2.164.67:42421)
I20260926 19:39:49.764411 8845 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:59869
I20260926 19:39:49.769495 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.164.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.125.keytab.
Entry for principal oryx/127.2.164.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.125.keytab.
I20260926 19:39:49.808786 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.125:35039
--webserver_interface=127.2.164.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.125.keytab
--principal=oryx/127.2.164.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:39835,127.2.164.125:35039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:39:49.857259 9034 heartbeater.cc:503] Master 127.2.164.126:39835 was elected leader, sending a full tablet report...
W20260926 19:39:50.106105 9306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:50.106351 9306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:50.106441 9306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:50.113302 9306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:50.113392 9306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:50.113440 9306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:50.113492 9306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:50.113539 9306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:50.122565 9306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:39835,127.2.164.125:35039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.125.keytab
--principal=oryx/127.2.164.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.125:35039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.164.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
W20260926 19:39:50.122826 9306 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.164.126:39835,127.2.164.125:35039'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 19:39:50.123736 9306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:50.124984 9306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:50.135604 9312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:50.135653 9311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:50.135653 9314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:50.136873 9306 server_base.cc:1034] running on GCE node
I20260926 19:39:50.137630 9306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:50.138892 9306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:50.140105 9306 hybrid_clock.cc:648] HybridClock initialized: now 1790451590140031 us; error 47 us; skew 500 ppm
I20260926 19:39:50.140398 9306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451590, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:50.143288 9306 init.cc:377] Logged in from keytab as oryx/127.2.164.125@KRBTEST.COM (short username oryx)
I20260926 19:39:50.144600 9306 webserver.cc:460] Webserver started at http://127.2.164.125:45305/ using document root <none> and password file <none>
I20260926 19:39:50.145264 9306 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:50.145359 9306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:50.145635 9306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:50.148386 9306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data/instance:
uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d"
format_stamp: "Formatted at 2026-09-26 19:39:50 on dist-test-slave-mrw4"
server_key: "82afbb15b499b3ebcdfe2ff68d45d6d2"
server_key_iv: "0bf276983753abb357c1c5462b1c41da"
server_key_version: "encryptionkey@0"
I20260926 19:39:50.149102 9306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/wal/instance:
uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d"
format_stamp: "Formatted at 2026-09-26 19:39:50 on dist-test-slave-mrw4"
server_key: "82afbb15b499b3ebcdfe2ff68d45d6d2"
server_key_iv: "0bf276983753abb357c1c5462b1c41da"
server_key_version: "encryptionkey@0"
I20260926 19:39:50.153864 9306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 19:39:50.156973 9321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:50.158335 9306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:39:50.158545 9306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/wal
uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d"
format_stamp: "Formatted at 2026-09-26 19:39:50 on dist-test-slave-mrw4"
server_key: "82afbb15b499b3ebcdfe2ff68d45d6d2"
server_key_iv: "0bf276983753abb357c1c5462b1c41da"
server_key_version: "encryptionkey@0"
I20260926 19:39:50.158754 9306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:50.193372 9306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:50.197337 9306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:50.197594 9306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:50.214262 9306 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.125:35039
I20260926 19:39:50.214294 9372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.125:35039 every 8 connection(s)
I20260926 19:39:50.215520 9306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
I20260926 19:39:50.218986 9373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:50.224241 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 9306
I20260926 19:39:50.225028 9373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451590, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.125@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260926 19:39:50.242762 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.227675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.2.164.126:39835)
0926 19:39:50.227853 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.227858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.228576 (+ 718us) server_negotiation.cc:407] Connection header received
0926 19:39:50.229024 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.229029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.229109 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.229251 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.230566 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.231429 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.233155 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.233786 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.233884 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.237252 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.237259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.237262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.237291 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.239815 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.240656 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.240661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.240665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.240792 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.241205 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.241210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.241214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.241488 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.241606 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.242247 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.242445 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:39:50.243650 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.227709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56408)
0926 19:39:50.228442 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.228478 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.228812 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.229317 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.229324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.230005 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.230414 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.230429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.231548 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.231554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.233024 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.233035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.233965 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.233971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.234223 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.234915 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.234936 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.237048 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.239955 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.239965 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.239975 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.240549 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.240896 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.240901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.240905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.241061 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.241612 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.241619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.241892 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.242143 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.242557 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":335,"threads_started":1}
I20260926 19:39:50.244120 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.226243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.2.164.125:35039)
0926 19:39:50.226915 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.226925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.226962 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:39:50.228815 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.228835 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.229091 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.229302 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.230913 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.232412 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.233815 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.234544 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.234635 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.238470 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.238480 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.238493 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.238533 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.241265 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.241865 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.241870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.241873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.241992 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.242334 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.242338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.242340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.242651 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.242921 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.243616 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.243793 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":300,"threads_started":1}
I20260926 19:39:50.245297 9373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451590, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.125@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260926 19:39:50.256067 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.246021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35039 (local address 127.0.0.1:59632)
0926 19:39:50.246264 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.246276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.246361 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.246593 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.246597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.246882 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.246989 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.246995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.247867 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.247871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.248681 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.248690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.249312 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.249317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.249499 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.249881 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.249894 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.251601 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.254061 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.254064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.254067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.254401 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.254699 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.254702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.254704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.254813 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.255234 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.255237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.255348 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.255457 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.255691 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":3}
I20260926 19:39:50.256124 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.246052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59632 (local address 127.2.164.125:35039)
0926 19:39:50.246206 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.246211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.246332 (+ 121us) server_negotiation.cc:407] Connection header received
0926 19:39:50.246444 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.246449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.246503 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.246569 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.247100 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.247747 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.248792 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.249199 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.249302 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.251750 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.251754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.251757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.251783 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.253940 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.254492 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.254496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.254498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.254611 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.254891 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.254894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.254896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.255139 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.255235 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.255551 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.255822 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":49}
I20260926 19:39:50.260347 9373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: Bootstrap starting.
I20260926 19:39:50.263298 9373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:50.264202 9373 log.cc:824] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:50.266625 9373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: No bootstrap required, opened a new log
I20260926 19:39:50.273073 9373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } }
I20260926 19:39:50.273401 9373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:50.273480 9373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e76dcb9e13f4dc2a2bdd4eda6ebe25d, State: Initialized, Role: FOLLOWER
I20260926 19:39:50.274058 9373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } }
I20260926 19:39:50.274858 9379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } } }
I20260926 19:39:50.275180 9379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:39:50.275889 9373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: configured and running, proceeding with master startup.
W20260926 19:39:50.283111 9390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:39:50.283221 9390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:39:50.285769 9306 master_runner.cc:428] Detected that this master 7e76dcb9e13f4dc2a2bdd4eda6ebe25d is joining an existing cluster
I20260926 19:39:50.285854 9306 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.164.125:35039
I20260926 19:39:50.301057 9168 heartbeater.cc:503] Master 127.2.164.126:39835 was elected leader, sending a full tablet report...
I20260926 19:39:50.311807 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.292190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.2.164.126:39835)
0926 19:39:50.292525 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.292530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.297847 (+ 5317us) server_negotiation.cc:407] Connection header received
0926 19:39:50.298093 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.298099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.298177 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.298245 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.299610 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.300607 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.302248 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.302793 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.302876 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.305479 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.305485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.305490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.305524 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.308552 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.309263 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.309269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.309273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.309631 (+ 358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.310046 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.310051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.310054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.310399 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.310509 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.311079 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.311271 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260926 19:39:50.313316 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.291597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56414)
0926 19:39:50.297747 (+ 6150us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.297766 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.297944 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.298439 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.298443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.299096 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.299450 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.299462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.300762 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.300766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.302105 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.302118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.303132 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.303138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.303286 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.303927 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.303947 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.305287 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.308702 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.308707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.308710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.309123 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.309764 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.309769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.309773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.309935 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.310577 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.310581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.310775 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.312287 (+ 1512us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.312826 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6012,"thread_start_us":11236,"threads_started":1}
I20260926 19:39:50.317133 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.292029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.2.164.125:35039)
0926 19:39:50.292388 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.292394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.302851 (+ 10457us) server_negotiation.cc:407] Connection header received
0926 19:39:50.303336 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.303341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.303412 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.303476 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.304872 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.305988 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.307842 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.308534 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.308620 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.310812 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.310818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.310826 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.310858 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.314390 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.315070 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.315075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.315079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.315227 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.315632 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.315636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.315640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.315965 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.316095 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.316586 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.316791 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260926 19:39:50.317838 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.291674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35039 (local address 127.0.0.1:59634)
0926 19:39:50.302765 (+ 11091us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.302788 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.303199 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.303645 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.303650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.304400 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.304725 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.304737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.306203 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.306208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.307703 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.307716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.308886 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.308892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.309061 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.309666 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.309691 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.310641 (+ 950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.314516 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.314520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.314524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.314949 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.315355 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.315359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.315363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.315521 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.316184 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.316189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.316329 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.317132 (+ 803us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.317415 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10982,"spinlock_wait_cycles":25984,"thread_start_us":12624,"threads_started":1}
WARNING: no policy specified for oryx/127.2.164.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.124@KRBTEST.COM" created.
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451588, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.126@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260926 19:39:50.334139 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.319737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.2.164.125:35039)
0926 19:39:50.319883 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.319888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.321896 (+ 2008us) server_negotiation.cc:407] Connection header received
0926 19:39:50.322155 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.322159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.322236 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.322340 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.323339 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.324139 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.325407 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.325848 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.325947 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.328712 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.328717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.328720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.328746 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.331702 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.332304 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.332309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.332312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.332448 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.332770 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.332775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.332779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.333017 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.333138 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.333596 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.333791 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:39:50.334183 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.319727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35039 (local address 127.0.0.1:59638)
0926 19:39:50.321673 (+ 1946us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.321750 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.321952 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.322405 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.322413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.322982 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.323170 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.323179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.324255 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.324260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.325293 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.325305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.325984 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.325990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.326150 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.326662 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.326678 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.328551 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.331802 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.331813 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.331821 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.332204 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.332542 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.332547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.332551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.332677 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.333128 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.333136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.333339 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.333504 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.333789 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1715,"thread_start_us":1561,"threads_started":1}
I20260926 19:39:50.336035 8845 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.164.125:35039: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d"
member_type: NON_VOTER
last_known_addr {
host: "127.2.164.125"
port: 35039
}
attrs {
promote: true
}
}
dest_uuid: "1547d4ec04a04ec58ebfae894e02340f"
cas_config_opid_index: -1
I20260926 19:39:50.337090 8845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } }
I20260926 19:39:50.345289 9402 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7e76dcb9e13f4dc2a2bdd4eda6ebe25d (127.2.164.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } } }
I20260926 19:39:50.347018 9348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:50.349989 9402 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.164.125:35039
I20260926 19:39:50.350373 9348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 FOLLOWER]: Refusing update from remote peer 1547d4ec04a04ec58ebfae894e02340f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 19:39:50.350775 9306 master.cc:537] Master@127.2.164.125:35039 shutting down...
I20260926 19:39:50.350931 9403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } } }
I20260926 19:39:50.351586 9403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:50.351640 9402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.124.keytab.
Entry for principal oryx/127.2.164.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.124.keytab.
I20260926 19:39:50.354628 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.124:35195
--webserver_interface=127.2.164.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.124.keytab
--principal=oryx/127.2.164.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:39835,127.2.164.125:35039,127.2.164.124:35195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:50.355684 8832 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 19:39:50.361472 9306 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 19:39:50.361477 8832 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f -> Peer 7e76dcb9e13f4dc2a2bdd4eda6ebe25d (127.2.164.125:35039): Couldn't send request to peer 7e76dcb9e13f4dc2a2bdd4eda6ebe25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 19:39:50.361968 9306 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:50.362258 9306 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: stopping tablet replica
I20260926 19:39:50.368002 9306 master.cc:552] Master@127.2.164.125:35039 shutdown complete.
I20260926 19:39:50.368119 9306 master_runner.cc:305] Clearing existing system tablet
I20260926 19:39:50.368572 9306 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:50.368706 9306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:50.371520 9306 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260926 19:39:50.373158 9408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:50.373476 9306 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:50.373883 9306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 19:39:50.374001 9306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/wal
uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d"
format_stamp: "Formatted at 2026-09-26 19:39:50 on dist-test-slave-mrw4"
server_key: "82afbb15b499b3ebcdfe2ff68d45d6d2"
server_key_iv: "0bf276983753abb357c1c5462b1c41da"
server_key_version: "encryptionkey@0"
I20260926 19:39:50.379194 9306 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:50.385829 9306 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 19:39:50.386051 9306 log.cc:1196] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 19:39:50.386489 9306 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: Deleting consensus metadata
I20260926 19:39:50.386790 9306 master_runner.cc:315] Copying system tablet from 127.2.164.126:39835
I20260926 19:39:50.389109 9306 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: tablet copy: Beginning tablet copy session from remote peer at address 127.2.164.126:39835
I20260926 19:39:50.407240 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.389978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56426 (local address 127.2.164.126:39835)
0926 19:39:50.390153 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.390160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.393679 (+ 3519us) server_negotiation.cc:407] Connection header received
0926 19:39:50.393853 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.393859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.393955 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.394068 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.395044 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.396123 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.397901 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.398489 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.398627 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.400719 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.400725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.400730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.400773 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.404346 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.405083 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.405088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.405092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.405279 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.405685 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.405690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.405694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.406060 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.406221 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.406648 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.406887 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:39:50.407269 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.389910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56426)
0926 19:39:50.393537 (+ 3627us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.393561 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.393731 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.394103 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.394108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.394625 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.394889 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.394900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.396263 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.396269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.397744 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.397759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.398635 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.398641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.398833 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.399502 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.399524 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.400509 (+ 985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.404472 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.404478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.404482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.404952 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.405401 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.405406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.405410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.405575 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.406181 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.406186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.406357 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.406506 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.406827 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3498,"thread_start_us":3391,"threads_started":1}
I20260926 19:39:50.408433 8865 tablet_copy_service.cc:140] P 1547d4ec04a04ec58ebfae894e02340f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7e76dcb9e13f4dc2a2bdd4eda6ebe25d ({username='oryx', principal='oryx/127.2.164.125@KRBTEST.COM'} at 127.0.0.1:56426)
I20260926 19:39:50.408757 8865 tablet_copy_service.cc:161] P 1547d4ec04a04ec58ebfae894e02340f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7e76dcb9e13f4dc2a2bdd4eda6ebe25d at {username='oryx', principal='oryx/127.2.164.125@KRBTEST.COM'} at 127.0.0.1:56426: session id = 7e76dcb9e13f4dc2a2bdd4eda6ebe25d-00000000000000000000000000000000
I20260926 19:39:50.413122 8865 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 19:39:50.415197 9306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 19:39:50.421708 9306 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: tablet copy: Starting download of 0 data blocks...
I20260926 19:39:50.421973 9306 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: tablet copy: Starting download of 1 WAL segments...
I20260926 19:39:50.424495 9306 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 19:39:50.428735 8865 tablet_copy_service.cc:342] P 1547d4ec04a04ec58ebfae894e02340f: Request end of tablet copy session 7e76dcb9e13f4dc2a2bdd4eda6ebe25d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.164.125@KRBTEST.COM'} at 127.0.0.1:56426
I20260926 19:39:50.428916 8865 tablet_copy_service.cc:433] P 1547d4ec04a04ec58ebfae894e02340f: ending tablet copy session 7e76dcb9e13f4dc2a2bdd4eda6ebe25d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7e76dcb9e13f4dc2a2bdd4eda6ebe25d
I20260926 19:39:50.439677 9306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:50.442185 9415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:50.446509 9416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:50.454270 9306 server_base.cc:1034] running on GCE node
W20260926 19:39:50.454666 9418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:50.455113 9306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:50.455788 9306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:50.457027 9306 hybrid_clock.cc:648] HybridClock initialized: now 1790451590456972 us; error 33 us; skew 500 ppm
I20260926 19:39:50.457295 9306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451590, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:50.459411 9306 init.cc:377] Logged in from keytab as oryx/127.2.164.125@KRBTEST.COM (short username oryx)
I20260926 19:39:50.460628 9306 webserver.cc:460] Webserver started at http://127.2.164.125:46085/ using document root <none> and password file <none>
I20260926 19:39:50.460991 9306 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:50.461185 9306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:50.464057 9306 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 19:39:50.465934 9425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:50.466361 9306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 19:39:50.466519 9306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/wal
uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d"
format_stamp: "Formatted at 2026-09-26 19:39:50 on dist-test-slave-mrw4"
server_key: "82afbb15b499b3ebcdfe2ff68d45d6d2"
server_key_iv: "0bf276983753abb357c1c5462b1c41da"
server_key_version: "encryptionkey@0"
I20260926 19:39:50.466697 9306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:50.488031 9306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:50.491776 9306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:50.514142 9306 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.125:35039
I20260926 19:39:50.514165 9476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.125:35039 every 8 connection(s)
I20260926 19:39:50.515146 9306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
I20260926 19:39:50.520097 9477 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:39:50.523981 9477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: Bootstrap starting.
I20260926 19:39:50.544165 9477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:39:50.544616 9477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: Bootstrap complete.
I20260926 19:39:50.545605 9477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } }
I20260926 19:39:50.545859 9477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e76dcb9e13f4dc2a2bdd4eda6ebe25d, State: Initialized, Role: LEARNER
I20260926 19:39:50.546152 9477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } }
I20260926 19:39:50.546967 9480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } } }
I20260926 19:39:50.547178 9477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:50.547366 9480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: This master's current role is: LEARNER
I20260926 19:39:50.557181 9491 catalog_manager.cc:1259] Loaded cluster ID: cf759cf0b7094eb98ac9c84b7965c53e
I20260926 19:39:50.557279 9491 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: loading cluster ID for follower catalog manager: success
I20260926 19:39:50.560699 9491 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: acquiring CA information for follower catalog manager: success
I20260926 19:39:50.563277 9491 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 19:39:50.714154 9406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:50.714413 9406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:50.714534 9406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:50.721372 9406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:50.721458 9406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:50.721527 9406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:50.721585 9406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:50.721637 9406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:50.731133 9406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:39835,127.2.164.125:35039,127.2.164.124:35195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.124.keytab
--principal=oryx/127.2.164.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.124:35195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.164.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:50.732460 9406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:50.734146 9406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:50.746732 9499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:50.746841 9496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:50.746748 9497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:50.747325 9406 server_base.cc:1034] running on GCE node
I20260926 19:39:50.748027 9406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:50.749342 9406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:50.750524 9406 hybrid_clock.cc:648] HybridClock initialized: now 1790451590750475 us; error 41 us; skew 500 ppm
I20260926 19:39:50.750838 9406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451590, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:50.754015 9406 init.cc:377] Logged in from keytab as oryx/127.2.164.124@KRBTEST.COM (short username oryx)
I20260926 19:39:50.755275 9406 webserver.cc:460] Webserver started at http://127.2.164.124:46135/ using document root <none> and password file <none>
I20260926 19:39:50.755919 9406 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:50.756026 9406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:50.756278 9406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:50.758967 9406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data/instance:
uuid: "96d207b9c12c4630b062682fbe003611"
format_stamp: "Formatted at 2026-09-26 19:39:50 on dist-test-slave-mrw4"
server_key: "35c761e19e7cafd4ad90768f59716b75"
server_key_iv: "a971a08adc90badaa8dd4ed735b6f073"
server_key_version: "encryptionkey@0"
I20260926 19:39:50.759658 9406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/wal/instance:
uuid: "96d207b9c12c4630b062682fbe003611"
format_stamp: "Formatted at 2026-09-26 19:39:50 on dist-test-slave-mrw4"
server_key: "35c761e19e7cafd4ad90768f59716b75"
server_key_iv: "a971a08adc90badaa8dd4ed735b6f073"
server_key_version: "encryptionkey@0"
I20260926 19:39:50.764190 9406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 19:39:50.766541 9301 heartbeater.cc:503] Master 127.2.164.126:39835 was elected leader, sending a full tablet report...
I20260926 19:39:50.767467 9506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:50.768913 9406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:39:50.769047 9406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/wal
uuid: "96d207b9c12c4630b062682fbe003611"
format_stamp: "Formatted at 2026-09-26 19:39:50 on dist-test-slave-mrw4"
server_key: "35c761e19e7cafd4ad90768f59716b75"
server_key_iv: "a971a08adc90badaa8dd4ed735b6f073"
server_key_version: "encryptionkey@0"
I20260926 19:39:50.769249 9406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:50.782491 9406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:50.786162 9406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:50.786384 9406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:50.800417 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.792994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35039 (local address 127.0.0.1:59642)
0926 19:39:50.793223 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.793243 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.793360 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.794518 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.794523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.794552 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:50.794735 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.794746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.796722 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.796728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.798309 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.798325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.799696 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.799701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.799888 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.799945 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.800028 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.800158 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 19:39:50.800486 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.793134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.2.164.125:35039)
0926 19:39:50.794185 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.794190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.794207 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:39:50.794267 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.794270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.794336 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.794449 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:50.794910 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.796543 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.798453 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.799532 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.799648 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.799880 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.800070 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.800206 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":816,"threads_started":1}
I20260926 19:39:50.802192 9452 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 19:39:50.802799 9480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1547d4ec04a04ec58ebfae894e02340f. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } } }
I20260926 19:39:50.803043 9480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: This master's current role is: LEARNER
I20260926 19:39:50.804966 9406 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.124:35195
I20260926 19:39:50.804987 9558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.124:35195 every 8 connection(s)
I20260926 19:39:50.806165 9406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
I20260926 19:39:50.806820 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 9406
I20260926 19:39:50.810612 9559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:50.816872 9559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451590, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.124@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:39:50.828379 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.808912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.2.164.124:35195)
0926 19:39:50.809601 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.809611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.809648 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:39:50.811126 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.811146 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.811490 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.811799 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.813260 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.814553 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.815721 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.816340 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.816486 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.820585 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.820591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.820601 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.820636 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.823601 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.824193 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.824201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.824207 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.824357 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.824754 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.824759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.824762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.825425 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.826557 (+ 1132us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.826806 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.827198 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":321,"threads_started":1}
I20260926 19:39:50.833061 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.819218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.2.164.126:39835)
0926 19:39:50.819371 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.819377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.820029 (+ 652us) server_negotiation.cc:407] Connection header received
0926 19:39:50.820179 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.820185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.820255 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.820329 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.821450 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.822542 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.823920 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.824346 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.824474 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.827475 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.827481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.827486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.827515 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.830197 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.831436 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.831441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.831444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.831597 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.831908 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.831911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.831913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.832171 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.832294 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.832659 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.832824 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:39:50.833079 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.818927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56428)
0926 19:39:50.819918 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.819954 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.820076 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.820397 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.820404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.821016 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.821297 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.821310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.822666 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.822669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.823808 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.823818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.824499 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.824505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.824664 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.825237 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.825257 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.827318 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.830342 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.830351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.830355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.831311 (+ 956us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.831697 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.831700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.831703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.831830 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.832304 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.832309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.832444 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.832590 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.832820 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":226,"threads_started":1}
I20260926 19:39:50.834369 9559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451590, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.124@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260926 19:39:50.845747 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.835119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35039 (local address 127.0.0.1:59650)
0926 19:39:50.835297 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.835314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.835399 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.835649 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.835651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.835925 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.836041 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.836048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.836963 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.836967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.837928 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.837939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.838634 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.838638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.838767 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.839271 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.839286 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.841279 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.843900 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.843904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.843906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.844264 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.844542 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.844545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.844547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.844663 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.845098 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.845101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.845242 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.845343 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.845546 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 19:39:50.845786 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.835246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.2.164.125:35039)
0926 19:39:50.835406 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.835410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.835428 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:39:50.835474 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.835478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.835542 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.835618 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.836171 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.836861 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.838054 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.838488 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.838614 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.841440 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.841445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.841448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.841476 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.843789 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.844345 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.844348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.844351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.844456 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.844762 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.844765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.844767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.845019 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.845122 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.845407 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.845562 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:39:50.846675 9559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451590, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.124@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
I20260926 19:39:50.857867 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.847386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:35195 (local address 127.0.0.1:55766)
0926 19:39:50.847571 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.847588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.847683 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.847941 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.847945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.848310 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.848423 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.848430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.849338 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.849344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.850164 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.850174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.850801 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.850807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.850988 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.851422 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.851437 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.853113 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.855857 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.855862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.855866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.856160 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.856465 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.856470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.856474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.856596 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.857002 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.857006 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.857132 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.857262 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.857483 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 19:39:50.857928 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.847422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.2.164.124:35195)
0926 19:39:50.847555 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.847560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.847642 (+ 82us) server_negotiation.cc:407] Connection header received
0926 19:39:50.847795 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.847800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.847849 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.847915 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.848547 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.849244 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.850282 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.850661 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.850759 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.853280 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.853286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.853290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.853316 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.855757 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.856266 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.856270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.856274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.856379 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.856679 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.856683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.856687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.856909 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.857021 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.857352 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.857609 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":36}
I20260926 19:39:50.863238 9559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: Bootstrap starting.
I20260926 19:39:50.866252 9559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:50.867251 9559 log.cc:824] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:50.869637 9559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: No bootstrap required, opened a new log
I20260926 19:39:50.876655 9559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } }
I20260926 19:39:50.876951 9559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:50.877050 9559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96d207b9c12c4630b062682fbe003611, State: Initialized, Role: FOLLOWER
I20260926 19:39:50.877600 9559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } }
I20260926 19:39:50.878412 9565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } } }
I20260926 19:39:50.878666 9565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:39:50.879295 9559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 19:39:50.887823 9576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:39:50.887939 9576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:39:50.891777 9406 master_runner.cc:428] Detected that this master 96d207b9c12c4630b062682fbe003611 is joining an existing cluster
I20260926 19:39:50.891861 9406 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.164.124:35195
I20260926 19:39:50.921099 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.895529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56434)
0926 19:39:50.906165 (+ 10636us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.906186 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.906656 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.907180 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.907185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.908692 (+ 1507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.908969 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.908981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.910268 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.910273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.911502 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.911513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.912557 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.912564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.912725 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.913456 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.913478 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.914682 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.918227 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.918233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.918240 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.918733 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.919168 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.919176 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.919180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.919357 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.919988 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.919993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.920204 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.920353 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.920626 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10484,"thread_start_us":10775,"threads_started":1}
I20260926 19:39:50.921839 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.895891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.2.164.126:39835)
0926 19:39:50.896032 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.896036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.906280 (+ 10244us) server_negotiation.cc:407] Connection header received
0926 19:39:50.906815 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.906822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.906904 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.906973 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.909100 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.910133 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.911625 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.912234 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.912345 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.914893 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.914899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.914904 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.914940 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.918070 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.918876 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.918881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.918885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.919048 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.919481 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.919486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.919488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.919808 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.919933 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.921416 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.921600 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:39:50.926972 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.895515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.2.164.124:35195)
0926 19:39:50.895958 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.895962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.909857 (+ 13895us) server_negotiation.cc:407] Connection header received
0926 19:39:50.910028 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.910032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.910085 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.910137 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.913299 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.914359 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.916740 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.917366 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.917457 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.919690 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.919696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.919701 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.919730 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.924144 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.924903 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.924908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.924911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.925083 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.925487 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.925491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.925495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.925850 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.925990 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.926474 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.926650 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289}
I20260926 19:39:50.927281 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.895502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.2.164.125:35039)
0926 19:39:50.895673 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.895679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.910137 (+ 14458us) server_negotiation.cc:407] Connection header received
0926 19:39:50.910401 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.910406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.910483 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.910577 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.913316 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.914414 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.916962 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.917622 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.917716 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.920063 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.920069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.920074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.920111 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.923695 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.925036 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.925042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.925045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.925212 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.925674 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.925678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.925682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.925990 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.926110 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.926721 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.926940 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:39:50.927835 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.896224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:35195 (local address 127.0.0.1:55768)
0926 19:39:50.909772 (+ 13548us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.909788 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.909931 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.910292 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.910296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.912708 (+ 2412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.913134 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.913145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.914560 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.914564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.916594 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.916610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.917708 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.917714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.917869 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.918438 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.918459 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.919534 (+ 1075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.924283 (+ 4749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.924288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.924291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.924796 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.925237 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.925241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.925244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.925377 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.926065 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.926068 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.926220 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.927151 (+ 931us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.927410 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13337,"mutex_wait_us":49,"thread_start_us":12103,"threads_started":1}
I20260926 19:39:50.928277 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.896398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35039 (local address 127.0.0.1:59662)
0926 19:39:50.910038 (+ 13640us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.910058 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.910239 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.910626 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.910632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.912690 (+ 2058us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.913118 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.913131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.914617 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.914622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.916801 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.916815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.917996 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.918002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.918173 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.918752 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.918774 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.919862 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.924339 (+ 4477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.924343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.924347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.924902 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.925371 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.925376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.925380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.925548 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.926202 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.926206 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.926449 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.927515 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.927798 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13524,"mutex_wait_us":65,"thread_start_us":13339,"threads_started":1}
I20260926 19:39:50.934379 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 8906
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451588, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.126@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
I20260926 19:39:50.942920 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.927545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:35195 (local address 127.0.0.1:55776)
0926 19:39:50.927839 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:50.927856 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:50.927948 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:50.929247 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:50.929251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:50.929632 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:50.929833 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.929844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.930925 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.930932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.932300 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:50.932314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.932973 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.932980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:50.933146 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.934084 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:50.934106 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:50.937050 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:50.940791 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.940796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.940801 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.941140 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.941464 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:50.941469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:50.941473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:50.941595 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:50.942046 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:50.942050 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:50.942188 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 19:39:50.942342 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 19:39:50.942588 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260926 19:39:50.943761 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.928136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.2.164.124:35195)
0926 19:39:50.928884 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.928888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.928904 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:39:50.928970 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.928975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.929069 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.929949 (+ 880us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.930074 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.930783 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.932422 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.932836 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.932924 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.937255 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.937261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.937266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.937300 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.940670 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.941260 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.941265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.941269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.941373 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.941684 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.941688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.941692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.941943 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.942063 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.943244 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.943436 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652}
I20260926 19:39:50.945010 8845 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.164.124:35195: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "96d207b9c12c4630b062682fbe003611"
member_type: NON_VOTER
last_known_addr {
host: "127.2.164.124"
port: 35195
}
attrs {
promote: true
}
}
dest_uuid: "1547d4ec04a04ec58ebfae894e02340f"
cas_config_opid_index: 5
Sep 26 19:39:50 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.65@KRBTEST.COM
I20260926 19:39:50.945892 8845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } }
I20260926 19:39:50.947431 9402 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 96d207b9c12c4630b062682fbe003611 (127.2.164.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:50.948195 9534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:50.950810 9402 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.164.124:35195
I20260926 19:39:50.951835 9452 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 LEARNER]: Refusing update from remote peer 1547d4ec04a04ec58ebfae894e02340f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 19:39:50.954491 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:50.935261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.2.164.65:45559)
0926 19:39:50.935824 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:50.935831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:50.935854 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:39:50.935903 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:50.935909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:50.936105 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:50.936249 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:50.937562 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.939086 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.940702 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:50.941595 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:50.942241 (+ 646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:50.946694 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:50.946719 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:50.946735 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:50.946775 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:50.950373 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.951205 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.951214 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.951217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.951361 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:50.951811 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:50.951816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:50.951819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:50.952293 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:50.952599 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:50.953777 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0926 19:39:50.954051 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":285,"threads_started":1}
I20260926 19:39:50.956117 9029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:50.963771 9402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 19:39:50.969627 9586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:50.969967 9586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:50.971051 9452 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 96d207b9c12c4630b062682fbe003611 (127.2.164.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:50.973703 9534 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 FOLLOWER]: Refusing update from remote peer 1547d4ec04a04ec58ebfae894e02340f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 19:39:50.974193 9406 master.cc:537] Master@127.2.164.124:35195 shutting down...
I20260926 19:39:50.975337 9587 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:50.987884 9406 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:50.988294 9406 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:50.988485 9406 tablet_replica.cc:330] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: stopping tablet replica
I20260926 19:39:51.001478 9406 master.cc:552] Master@127.2.164.124:35195 shutdown complete.
I20260926 19:39:51.001566 9406 master_runner.cc:305] Clearing existing system tablet
I20260926 19:39:51.001938 9406 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:51.002039 9406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:51.007155 9480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: NON_VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: true } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:51.007453 9480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: This master's current role is: LEARNER
I20260926 19:39:51.008486 9587 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f: attempting to promote NON_VOTER 7e76dcb9e13f4dc2a2bdd4eda6ebe25d to VOTER
I20260926 19:39:51.009346 9587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } }
I20260926 19:39:51.010308 9406 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
W20260926 19:39:51.012060 8831 connection.cc:698] client connection to 127.2.164.124:35195 send error: Network error: sendmsg error: Broken pipe (error 32)
I20260926 19:39:51.013900 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.013480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:35195 (local address 127.0.0.1:55782)
0926 19:39:51.013635 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:51.013741 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:35195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 19:39:51.014155 8831 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f -> Peer 96d207b9c12c4630b062682fbe003611 (127.2.164.124:35195): Couldn't send request to peer 96d207b9c12c4630b062682fbe003611. Status: Network error: Client connection negotiation failed: client connection to 127.2.164.124:35195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 19:39:51.017030 9452 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 LEARNER]: Refusing update from remote peer 1547d4ec04a04ec58ebfae894e02340f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 19:39:51.017598 9586 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 19:39:51.018200 9594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:51.019964 9587 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7e76dcb9e13f4dc2a2bdd4eda6ebe25d (127.2.164.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:51.020577 9451 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7e76dcb9e13f4dc2a2bdd4eda6ebe25d (127.2.164.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:51.023872 9402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:51.024161 9402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:51.024477 9402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:51.024717 9402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:51.024648 9480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:51.024893 9480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:39:51.025751 9406 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:51.026014 9406 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.001s sys 0.003s
I20260926 19:39:51.026120 9406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/wal
uuid: "96d207b9c12c4630b062682fbe003611"
format_stamp: "Formatted at 2026-09-26 19:39:50 on dist-test-slave-mrw4"
server_key: "35c761e19e7cafd4ad90768f59716b75"
server_key_iv: "a971a08adc90badaa8dd4ed735b6f073"
server_key_version: "encryptionkey@0"
I20260926 19:39:51.031106 9406 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:51.037809 9406 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 19:39:51.037992 9406 log.cc:1196] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 19:39:51.038419 9406 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: Deleting consensus metadata
I20260926 19:39:51.038667 9406 master_runner.cc:315] Copying system tablet from 127.2.164.126:39835
I20260926 19:39:51.041263 9406 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: tablet copy: Beginning tablet copy session from remote peer at address 127.2.164.126:39835
I20260926 19:39:51.060880 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.042032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.2.164.126:39835)
0926 19:39:51.042248 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:51.042254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:51.047439 (+ 5185us) server_negotiation.cc:407] Connection header received
0926 19:39:51.047638 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:51.047644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:51.047727 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:51.047828 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:51.048912 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.050080 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.051918 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.052551 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.052672 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:51.054778 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:51.054785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:51.054790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:51.054829 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:51.058154 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:51.058867 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:51.058873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:51.058876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:51.059023 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:51.059402 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:51.059407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:51.059411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:51.059744 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:51.059892 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:51.060298 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 19:39:51.060510 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 19:39:51.060921 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.042021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56440)
0926 19:39:51.047311 (+ 5290us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:51.047335 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:51.047489 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:51.047867 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:51.047872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:51.048502 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:51.048759 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:51.048771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.050268 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.050274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:51.051772 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:51.051787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.052710 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.052716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:51.052894 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:51.053593 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:51.053617 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:51.054572 (+ 955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:51.058291 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:51.058296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:51.058300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:51.058760 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:51.059134 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:51.059139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:51.059142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:51.059305 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:51.059863 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:51.059868 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:51.060050 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 19:39:51.060191 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 19:39:51.060490 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5137,"thread_start_us":4978,"threads_started":1}
I20260926 19:39:51.061594 8865 tablet_copy_service.cc:140] P 1547d4ec04a04ec58ebfae894e02340f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 96d207b9c12c4630b062682fbe003611 ({username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:56440)
I20260926 19:39:51.061764 8865 tablet_copy_service.cc:161] P 1547d4ec04a04ec58ebfae894e02340f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 96d207b9c12c4630b062682fbe003611 at {username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:56440: session id = 96d207b9c12c4630b062682fbe003611-00000000000000000000000000000000
I20260926 19:39:51.065780 8865 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 19:39:51.067664 9406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 19:39:51.074231 9406 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: tablet copy: Starting download of 0 data blocks...
I20260926 19:39:51.074501 9406 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: tablet copy: Starting download of 1 WAL segments...
I20260926 19:39:51.076871 9406 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 19:39:51.081501 8865 tablet_copy_service.cc:342] P 1547d4ec04a04ec58ebfae894e02340f: Request end of tablet copy session 96d207b9c12c4630b062682fbe003611-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:56440
I20260926 19:39:51.081662 8865 tablet_copy_service.cc:433] P 1547d4ec04a04ec58ebfae894e02340f: ending tablet copy session 96d207b9c12c4630b062682fbe003611-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 96d207b9c12c4630b062682fbe003611
I20260926 19:39:51.085180 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 8906
I20260926 19:39:51.092478 9406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:51.094892 9601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:51.102026 9602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:51.103370 9604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:51.103667 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:45559
--local_ip_for_outbound_sockets=127.2.164.65
--tserver_master_addrs=127.2.164.126:39835
--webserver_port=36699
--webserver_interface=127.2.164.65
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab
--principal=oryx/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.164.126:39835,127.2.164.125:35039
--tserver_master_addrs=127.2.164.126:39835,127.2.164.125:35039,127.2.164.124:35195 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:39:51.110481 9406 server_base.cc:1034] running on GCE node
I20260926 19:39:51.110808 9406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:51.111397 9406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:51.112592 9406 hybrid_clock.cc:648] HybridClock initialized: now 1790451591112549 us; error 38 us; skew 500 ppm
I20260926 19:39:51.112813 9406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:51 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451591, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:51.114863 9406 init.cc:377] Logged in from keytab as oryx/127.2.164.124@KRBTEST.COM (short username oryx)
I20260926 19:39:51.116083 9406 webserver.cc:460] Webserver started at http://127.2.164.124:37543/ using document root <none> and password file <none>
I20260926 19:39:51.116369 9406 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:51.116453 9406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:51.119379 9406 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:51.121240 9612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:51.121661 9406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 19:39:51.121821 9406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/wal
uuid: "96d207b9c12c4630b062682fbe003611"
format_stamp: "Formatted at 2026-09-26 19:39:50 on dist-test-slave-mrw4"
server_key: "35c761e19e7cafd4ad90768f59716b75"
server_key_iv: "a971a08adc90badaa8dd4ed735b6f073"
server_key_version: "encryptionkey@0"
I20260926 19:39:51.122074 9406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:51.134464 9406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:51.138258 9406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:51.165408 9406 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.124:35195
I20260926 19:39:51.165436 9663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.124:35195 every 8 connection(s)
I20260926 19:39:51.166651 9406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
I20260926 19:39:51.172185 9664 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:39:51.175792 9664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: Bootstrap starting.
I20260926 19:39:51.193185 9664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:39:51.193672 9664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: Bootstrap complete.
I20260926 19:39:51.194648 9664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } }
I20260926 19:39:51.194929 9664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 96d207b9c12c4630b062682fbe003611, State: Initialized, Role: LEARNER
I20260926 19:39:51.195176 9664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } }
I20260926 19:39:51.196218 9664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:51.196048 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:51.196651 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: This master's current role is: LEARNER
I20260926 19:39:51.207124 9678 catalog_manager.cc:1259] Loaded cluster ID: cf759cf0b7094eb98ac9c84b7965c53e
I20260926 19:39:51.207238 9678 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: loading cluster ID for follower catalog manager: success
I20260926 19:39:51.210712 9678 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: acquiring CA information for follower catalog manager: success
I20260926 19:39:51.213030 9678 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 19:39:51.439952 9606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:51.440228 9606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:51.440357 9606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:51.447305 9606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:51.447395 9606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:51.447530 9606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:51.460700 9606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab
--principal=oryx/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:45559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=36699
--tserver_master_addrs=127.2.164.126:39835,127.2.164.125:35039,127.2.164.124:35195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:51.462225 9606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:51.463779 9606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 19:39:51.474180 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.468226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:35195 (local address 127.0.0.1:55794)
0926 19:39:51.468388 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:51.468406 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:51.468529 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:51.469434 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:51.469439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:51.469484 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:51.469648 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:51.469657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.471142 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.471148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:51.472340 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:51.472353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.473437 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.473442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:51.473622 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:51.473657 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:39:51.473745 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 19:39:51.473875 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 19:39:51.474192 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.468650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.2.164.124:35195)
0926 19:39:51.469102 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:51.469106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:51.469122 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:39:51.469210 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:51.469214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:51.469284 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:51.469402 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:51.469807 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.470996 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.472489 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.473282 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.473399 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:51.473601 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:51.473782 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 19:39:51.473902 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":227,"threads_started":1}
I20260926 19:39:51.475229 9639 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 19:39:51.475888 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1547d4ec04a04ec58ebfae894e02340f. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:51.476146 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: This master's current role is: LEARNER
W20260926 19:39:51.478575 9684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:51.479351 9685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:51.482226 9687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:51.482817 9606 server_base.cc:1034] running on GCE node
I20260926 19:39:51.483354 9606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:51.484181 9606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:51.485368 9606 hybrid_clock.cc:648] HybridClock initialized: now 1790451591485323 us; error 36 us; skew 500 ppm
I20260926 19:39:51.485648 9606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:51 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451591, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:51.488425 9606 init.cc:377] Logged in from keytab as oryx/127.2.164.65@KRBTEST.COM (short username oryx)
I20260926 19:39:51.489862 9606 webserver.cc:460] Webserver started at http://127.2.164.65:36699/ using document root <none> and password file <none>
I20260926 19:39:51.490583 9606 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:51.490706 9606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:51.496074 9606 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:39:51.498612 9694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:51.499835 9606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 19:39:51.499970 9606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "b82043831211489bbc3c7f33beb81c91"
format_stamp: "Formatted at 2026-09-26 19:39:48 on dist-test-slave-mrw4"
server_key: "fb20b0d00d0086849708c0f1f68de6e4"
server_key_iv: "785d3e596dc2ec8ece28dd0e2f43ab7d"
server_key_version: "encryptionkey@0"
I20260926 19:39:51.500854 9606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:51.529592 9606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:51.533856 9606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:51.534085 9606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:51.534942 9606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:51.536188 9606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:51.536285 9606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:51.536378 9606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:51.536445 9606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:51.566606 9606 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:45559
I20260926 19:39:51.566646 9806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:45559 every 8 connection(s)
I20260926 19:39:51.567983 9606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:51.576260 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 9606
I20260926 19:39:51.576573 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 9040
Sep 26 19:39:51 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451591, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
Sep 26 19:39:51 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451591, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
Sep 26 19:39:51 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451591, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
I20260926 19:39:51.594753 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.570532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:34033 (local address 127.2.164.126:39835)
0926 19:39:51.570930 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:51.570935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:51.572422 (+ 1487us) server_negotiation.cc:407] Connection header received
0926 19:39:51.575335 (+ 2913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:51.575342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:51.575423 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:51.575498 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:51.577940 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.578997 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.581698 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.582369 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.582463 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:51.587334 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:51.587341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:51.587346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:51.587384 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:51.590691 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:51.591725 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:51.591731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:51.591735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:51.592008 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:51.592557 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:51.592563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:51.592566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:51.592916 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:51.593050 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:51.594139 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0926 19:39:51.594347 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":205,"threads_started":1}
I20260926 19:39:51.595211 9812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.570567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:45575 (local address 127.2.164.125:35039)
0926 19:39:51.573288 (+ 2721us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:51.573294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:51.574306 (+ 1012us) server_negotiation.cc:407] Connection header received
0926 19:39:51.575399 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:51.575404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:51.575491 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:51.575592 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:51.577820 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.578976 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.581719 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.582967 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.583087 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:51.588528 (+ 5441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:51.588535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:51.588541 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:51.588584 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:51.592585 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:51.593406 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:51.593411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:51.593415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:51.593552 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:51.593914 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:51.593919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:51.593923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:51.594161 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:51.594301 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:51.594645 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 19:39:51.594839 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2614,"thread_start_us":2502,"threads_started":1}
I20260926 19:39:51.596760 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.573416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35039 (local address 127.2.164.65:45575)
0926 19:39:51.574187 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:51.574213 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:51.575063 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:51.575788 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:51.575792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:51.577292 (+ 1500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:51.577624 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:51.577637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.579151 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.579157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:51.581523 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:51.581545 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.583367 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.583374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:51.583918 (+ 544us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:51.584551 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:51.584574 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:51.588285 (+ 3711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:51.592740 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:51.592744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:51.592747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:51.593270 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:51.593663 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:51.593668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:51.593672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:51.593807 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:51.594275 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:51.594280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:51.594411 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 19:39:51.594549 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 19:39:51.595035 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":143,"thread_start_us":505,"threads_started":1}
I20260926 19:39:51.596786 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.571245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.65:34033)
0926 19:39:51.572107 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:51.572311 (+ 204us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:51.575053 (+ 2742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:51.575743 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:51.575748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:51.576603 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:51.577787 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:51.577799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.579154 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.579159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:51.581531 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:51.581547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.582783 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.582796 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:51.583094 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:51.583886 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:51.583917 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:51.587089 (+ 3172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:51.590898 (+ 3809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:51.590908 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:51.590927 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:51.591585 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:51.592163 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:51.592172 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:51.592176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:51.592422 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:51.593142 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:51.593151 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:51.593702 (+ 551us) client_negotiation.cc:769] Sending connection context
0926 19:39:51.595253 (+ 1551us) client_negotiation.cc:240] Negotiation successful
0926 19:39:51.595664 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":145,"thread_start_us":1563,"threads_started":1}
I20260926 19:39:51.598475 9808 heartbeater.cc:347] Connected to a master server at 127.2.164.126:39835
I20260926 19:39:51.598481 9809 heartbeater.cc:347] Connected to a master server at 127.2.164.125:35039
I20260926 19:39:51.598821 9809 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:51.598817 9808 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:51.599895 9809 heartbeater.cc:511] Master 127.2.164.125:35039 requested a full tablet report, sending...
I20260926 19:39:51.599918 9808 heartbeater.cc:511] Master 127.2.164.126:39835 requested a full tablet report, sending...
I20260926 19:39:51.601565 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.572076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:52091 (local address 127.2.164.124:35195)
0926 19:39:51.572225 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:51.572232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:51.573298 (+ 1066us) server_negotiation.cc:407] Connection header received
0926 19:39:51.575318 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:51.575323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:51.575405 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:51.575490 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:51.577857 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.578986 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.589103 (+ 10117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.589732 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.589845 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:51.595371 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:51.595377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:51.595382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:51.595421 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:51.598643 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:51.599460 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:51.599465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:51.599468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:51.599608 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:51.600059 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:51.600064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:51.600067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:51.600392 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:51.600514 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:51.601018 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 19:39:51.601252 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 19:39:51.601748 8845 ts_manager.cc:194] Re-registered known tserver with Master: b82043831211489bbc3c7f33beb81c91 (127.2.164.65:45559)
I20260926 19:39:51.602428 9814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.571976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:35195 (local address 127.2.164.65:52091)
0926 19:39:51.573184 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:51.573202 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:51.575060 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:51.575732 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:51.575743 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:51.576655 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:51.577484 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:51.577508 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.585972 (+ 8464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.585975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:51.587391 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:51.587403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.590163 (+ 2760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.590169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:51.590338 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:51.591190 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:51.591210 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:51.594523 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:51.598789 (+ 4266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:51.598793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:51.598797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:51.599328 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:51.599767 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:51.599771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:51.599775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:51.599939 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:51.600591 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:51.600596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:51.600731 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 19:39:51.601802 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0926 19:39:51.602064 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":48,"spinlock_wait_cycles":15616,"thread_start_us":858,"threads_started":1}
I20260926 19:39:51.602605 8845 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:34033
I20260926 19:39:51.604089 9442 ts_manager.cc:194] Registered new tserver with Master: b82043831211489bbc3c7f33beb81c91 (127.2.164.65:45559)
I20260926 19:39:51.604594 9807 heartbeater.cc:347] Connected to a master server at 127.2.164.124:35195
I20260926 19:39:51.604741 9807 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:51.605077 9807 heartbeater.cc:511] Master 127.2.164.124:35195 requested a full tablet report, sending...
I20260926 19:39:51.606514 9629 ts_manager.cc:194] Registered new tserver with Master: b82043831211489bbc3c7f33beb81c91 (127.2.164.65:45559)
Sep 26 19:39:51 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.66@KRBTEST.COM
I20260926 19:39:51.614084 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:51.577584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.2.164.66:34565)
0926 19:39:51.597954 (+ 20370us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:51.597963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:51.597992 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:39:51.598062 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:51.598071 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:51.598323 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:51.600575 (+ 2252us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:51.600822 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.602305 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.604193 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:51.605085 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:51.605301 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:51.609525 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:51.609534 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:51.609547 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:51.609580 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:51.611830 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:51.612408 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:51.612413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:51.612415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:51.612518 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:51.612809 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:51.612813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:51.612815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:51.613108 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:51.613362 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:51.613608 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 19:39:51.613789 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20200,"thread_start_us":20864,"threads_started":1}
I20260926 19:39:51.615293 9163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:51.694713 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 9040
I20260926 19:39:51.707548 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:34565
--local_ip_for_outbound_sockets=127.2.164.66
--tserver_master_addrs=127.2.164.126:39835
--webserver_port=36407
--webserver_interface=127.2.164.66
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab
--principal=oryx/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.164.126:39835,127.2.164.125:35039
--tserver_master_addrs=127.2.164.126:39835,127.2.164.125:35039,127.2.164.124:35195 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:39:51.871421 9587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: NON_VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: true } } }
I20260926 19:39:51.871775 9587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:51.876417 9821 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f: attempting to promote NON_VOTER 96d207b9c12c4630b062682fbe003611 to VOTER
I20260926 19:39:51.877197 9821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } }
I20260926 19:39:51.879032 9639 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 LEARNER]: Refusing update from remote peer 1547d4ec04a04ec58ebfae894e02340f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 19:39:51.879108 9451 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 FOLLOWER]: Refusing update from remote peer 1547d4ec04a04ec58ebfae894e02340f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 19:39:51.879621 9821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [LEADER]: Connected to new peer: Peer: permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 19:39:51.879995 9820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 19:39:51.882477 9821 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96d207b9c12c4630b062682fbe003611 (127.2.164.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:51.883097 9451 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96d207b9c12c4630b062682fbe003611 (127.2.164.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:51.886605 9821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:51.886951 9821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:51.887221 9821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:51.887449 9821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:51.892140 9824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:51.893306 9824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:39:51.897307 9825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:51.897640 9825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:51.897840 9639 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 96d207b9c12c4630b062682fbe003611 (127.2.164.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:51.900533 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1547d4ec04a04ec58ebfae894e02340f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:51.900787 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 19:39:52.017366 9819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:52.017611 9819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:52.017726 9819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:52.024776 9819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:52.024871 9819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:52.024984 9819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:52.034337 9819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab
--principal=oryx/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:34565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=36407
--tserver_master_addrs=127.2.164.126:39835,127.2.164.125:35039,127.2.164.124:35195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:52.035529 9819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:52.036764 9819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:52.049189 9830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:52.049181 9831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:52.049216 9833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:52.049825 9819 server_base.cc:1034] running on GCE node
I20260926 19:39:52.050410 9819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:52.051326 9819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:52.052510 9819 hybrid_clock.cc:648] HybridClock initialized: now 1790451592052462 us; error 40 us; skew 500 ppm
I20260926 19:39:52.052786 9819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:52 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451592, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:52.055648 9819 init.cc:377] Logged in from keytab as oryx/127.2.164.66@KRBTEST.COM (short username oryx)
I20260926 19:39:52.056893 9819 webserver.cc:460] Webserver started at http://127.2.164.66:36407/ using document root <none> and password file <none>
I20260926 19:39:52.057585 9819 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:52.057695 9819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:52.062630 9819 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:39:52.065062 9840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:52.066371 9819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:39:52.066519 9819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "a3a543749a3e4f2fa93f5de7075a88d3"
format_stamp: "Formatted at 2026-09-26 19:39:49 on dist-test-slave-mrw4"
server_key: "721d8eb9885d55e4e42ce7b93ebeb7e8"
server_key_iv: "2423ab013ff041cff1be87725278f514"
server_key_version: "encryptionkey@0"
I20260926 19:39:52.067401 9819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:52.090456 9819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:52.094535 9819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:52.094771 9819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:52.095683 9819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:52.097025 9819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:52.097110 9819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:52.097241 9819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:52.097312 9819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:52.126363 9819 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:34565
I20260926 19:39:52.126433 9952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:34565 every 8 connection(s)
I20260926 19:39:52.127821 9819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:52.134349 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 9819
I20260926 19:39:52.134634 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 9173
Sep 26 19:39:52 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451592, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
Sep 26 19:39:52 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451592, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
Sep 26 19:39:52 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.67@KRBTEST.COM
I20260926 19:39:52.192842 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.130627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:34635 (local address 127.2.164.126:39835)
0926 19:39:52.160240 (+ 29613us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.160247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.166642 (+ 6395us) server_negotiation.cc:407] Connection header received
0926 19:39:52.170535 (+ 3893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.170542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.170627 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.170709 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:52.173570 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.174632 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.177089 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.177680 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.177770 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.186009 (+ 8239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:52.186015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:52.186020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:52.186058 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:52.189532 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.190379 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.190384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.190388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.190517 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.191013 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.191018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.191021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.191362 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:52.191470 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.192284 (+ 814us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.192499 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29478,"thread_start_us":29372,"threads_started":1}
I20260926 19:39:52.192867 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.130586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:36255 (local address 127.2.164.124:35195)
0926 19:39:52.160567 (+ 29981us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.160573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.172329 (+ 11756us) server_negotiation.cc:407] Connection header received
0926 19:39:52.174976 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.174981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.175058 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.175129 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:52.177461 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.178588 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.180145 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.180776 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.180900 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.185342 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:52.185349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:52.185354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:52.185392 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:52.188994 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.190054 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.190059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.190063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.190193 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.190848 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.190854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.190857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.191200 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:52.191320 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.192283 (+ 963us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.192491 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29859,"thread_start_us":29779,"threads_started":1}
Sep 26 19:39:52 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451592, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260926 19:39:52.195268 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.131288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.66:34635)
0926 19:39:52.166392 (+ 35104us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:52.166456 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:52.169245 (+ 2789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:52.171250 (+ 2005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:52.171261 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:52.172417 (+ 1156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:52.173377 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.173401 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.174842 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.174847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.176938 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.176951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.178084 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.178091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.178409 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.180273 (+ 1864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:52.180301 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:52.185801 (+ 5500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:52.189700 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.189705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.189709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.190253 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.190651 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.190654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.190658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.190889 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.191572 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:52.191576 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:52.191843 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 19:39:52.193122 (+ 1279us) client_negotiation.cc:240] Negotiation successful
0926 19:39:52.193847 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34766,"mutex_wait_us":219,"spinlock_wait_cycles":10112,"thread_start_us":53688,"threads_started":1}
I20260926 19:39:52.195307 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.131289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:35195 (local address 127.2.164.66:36255)
0926 19:39:52.172232 (+ 40943us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:52.172255 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:52.174826 (+ 2571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:52.175359 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:52.175363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:52.176011 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:52.176301 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.176312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.178922 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.178927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.180008 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.180019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.181188 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.181195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.181377 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.181990 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:52.182018 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:52.184612 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:52.189179 (+ 4567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.189189 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.189205 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.189925 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.190339 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.190343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.190347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.190546 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.191406 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:52.191414 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:52.191849 (+ 435us) client_negotiation.cc:769] Sending connection context
0926 19:39:52.193108 (+ 1259us) client_negotiation.cc:240] Negotiation successful
0926 19:39:52.193705 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40793,"mutex_wait_us":54,"spinlock_wait_cycles":72832,"thread_start_us":45725,"threads_started":1}
I20260926 19:39:52.196731 9954 heartbeater.cc:347] Connected to a master server at 127.2.164.126:39835
I20260926 19:39:52.196794 9953 heartbeater.cc:347] Connected to a master server at 127.2.164.124:35195
I20260926 19:39:52.197039 9954 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:52.197045 9953 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:52.198139 9954 heartbeater.cc:511] Master 127.2.164.126:39835 requested a full tablet report, sending...
I20260926 19:39:52.198164 9953 heartbeater.cc:511] Master 127.2.164.124:35195 requested a full tablet report, sending...
I20260926 19:39:52.199717 8845 ts_manager.cc:194] Re-registered known tserver with Master: a3a543749a3e4f2fa93f5de7075a88d3 (127.2.164.66:34565)
I20260926 19:39:52.199952 9629 ts_manager.cc:194] Registered new tserver with Master: a3a543749a3e4f2fa93f5de7075a88d3 (127.2.164.66:34565)
I20260926 19:39:52.200416 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.137818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:55757 (local address 127.2.164.125:35039)
0926 19:39:52.165697 (+ 27879us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.165703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.185630 (+ 19927us) server_negotiation.cc:407] Connection header received
0926 19:39:52.185755 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.185761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.185851 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.185940 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:52.187069 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.188167 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.189960 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.190521 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.190612 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.194232 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:52.194239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:52.194244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:52.194297 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:52.197494 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.198277 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.198282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.198286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.198429 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.198881 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.198886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.198890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.199220 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:52.199359 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.199848 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.200044 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27759,"thread_start_us":27882,"threads_started":1}
I20260926 19:39:52.200742 8845 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:34635
I20260926 19:39:52.201351 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.136030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.2.164.67:42421)
0926 19:39:52.168535 (+ 32505us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.168554 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.168585 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:39:52.168658 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.168666 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.168927 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.169087 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:52.172975 (+ 3888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.174639 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.182841 (+ 8202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.183675 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.183864 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.189984 (+ 6120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:52.189999 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:52.190015 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:52.190059 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:52.195800 (+ 5741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.197682 (+ 1882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.197691 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.197694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.197835 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.198307 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.198312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.198315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.198717 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:52.198998 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.200691 (+ 1693us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.200931 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32323,"thread_start_us":33266,"threads_started":1}
I20260926 19:39:52.201375 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.137407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35039 (local address 127.2.164.66:55757)
0926 19:39:52.185432 (+ 48025us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:52.185498 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:52.185642 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:52.186109 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:52.186114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:52.186630 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:52.186901 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.186913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.188337 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.188342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.189813 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.189828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.190884 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.190891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.191047 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.191572 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:52.191591 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:52.194018 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:52.197606 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.197611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.197614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.198142 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.198570 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.198574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.198578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.198766 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.199435 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:52.199440 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:52.199579 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 19:39:52.200629 (+ 1050us) client_negotiation.cc:240] Negotiation successful
0926 19:39:52.200915 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47911,"thread_start_us":32173,"threads_started":1}
I20260926 19:39:52.202080 9955 heartbeater.cc:347] Connected to a master server at 127.2.164.125:35039
I20260926 19:39:52.202234 9955 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:52.203082 9296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:52.203377 9955 heartbeater.cc:511] Master 127.2.164.125:35039 requested a full tablet report, sending...
I20260926 19:39:52.204468 9442 ts_manager.cc:194] Registered new tserver with Master: a3a543749a3e4f2fa93f5de7075a88d3 (127.2.164.66:34565)
I20260926 19:39:52.284107 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 9173
I20260926 19:39:52.301239 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:42421
--local_ip_for_outbound_sockets=127.2.164.67
--tserver_master_addrs=127.2.164.126:39835
--webserver_port=36659
--webserver_interface=127.2.164.67
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab
--principal=oryx/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.164.126:39835,127.2.164.125:35039
--tserver_master_addrs=127.2.164.126:39835,127.2.164.125:35039,127.2.164.124:35195 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:39:52.605988 9808 heartbeater.cc:503] Master 127.2.164.126:39835 was elected leader, sending a full tablet report...
W20260926 19:39:52.607545 9966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:52.607842 9966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:52.607981 9966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:52.615012 9966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:52.615103 9966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:52.615264 9966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:52.624742 9966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab
--principal=oryx/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:42421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=36659
--tserver_master_addrs=127.2.164.126:39835,127.2.164.125:35039,127.2.164.124:35195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:52.626009 9966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:52.627228 9966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:52.638849 9972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:52.638825 9973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:52.638877 9975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:52.640057 9966 server_base.cc:1034] running on GCE node
I20260926 19:39:52.640606 9966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:52.641445 9966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:52.642635 9966 hybrid_clock.cc:648] HybridClock initialized: now 1790451592642594 us; error 32 us; skew 500 ppm
I20260926 19:39:52.642925 9966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:52 dist-test-slave-mrw4 krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451592, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:52.645778 9966 init.cc:377] Logged in from keytab as oryx/127.2.164.67@KRBTEST.COM (short username oryx)
I20260926 19:39:52.647063 9966 webserver.cc:460] Webserver started at http://127.2.164.67:36659/ using document root <none> and password file <none>
I20260926 19:39:52.647739 9966 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:52.647858 9966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:52.652910 9966 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:39:52.655516 9982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:52.656788 9966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:39:52.656935 9966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "5cf6d073c00548fc93ebea3723743c47"
format_stamp: "Formatted at 2026-09-26 19:39:49 on dist-test-slave-mrw4"
server_key: "2c935f333aba468152d4bfa12504c055"
server_key_iv: "ba565ab3e36a5520e0fe8892c9f720e5"
server_key_version: "encryptionkey@0"
I20260926 19:39:52.657904 9966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:52.672221 9966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:52.676115 9966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:52.676344 9966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:52.677197 9966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:39:52.678468 9966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:52.678560 9966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:52.678656 9966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:52.678726 9966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:52.706385 9966 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:42421
I20260926 19:39:52.706430 10094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:42421 every 8 connection(s)
I20260926 19:39:52.707744 9966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:52.717495 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 9966
Sep 26 19:39:52 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451592, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:39:52.741398 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.710432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:37323 (local address 127.2.164.126:39835)
0926 19:39:52.710840 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.710847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.721853 (+ 11006us) server_negotiation.cc:407] Connection header received
0926 19:39:52.724854 (+ 3001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.724860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.724946 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.725045 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:52.727859 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.728862 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.730422 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.730812 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.730926 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.734274 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:52.734282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:52.734287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:52.734336 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:52.737572 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.738754 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.738760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.738764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.739008 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.739414 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.739419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.739423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.739791 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:52.739922 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.740802 (+ 880us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.741020 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":1025,"threads_started":1}
Sep 26 19:39:52 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451592, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.125@KRBTEST.COM
I20260926 19:39:52.751147 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.711072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.67:37323)
0926 19:39:52.721684 (+ 10612us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:52.721739 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:52.724426 (+ 2687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:52.726001 (+ 1575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:52.726015 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:52.726800 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:52.727675 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.727701 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.729000 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.729006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.730282 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.730302 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.731024 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.731031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.731253 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.731855 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:52.731881 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:52.734011 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:52.738187 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.738192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.738204 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.738642 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.739097 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.739102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.739105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.739306 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.739908 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:52.739915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:52.740227 (+ 312us) client_negotiation.cc:769] Sending connection context
0926 19:39:52.740512 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 19:39:52.741024 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10270,"spinlock_wait_cycles":34176,"thread_start_us":13668,"threads_started":1}
I20260926 19:39:52.751587 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.710451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:33433 (local address 127.2.164.125:35039)
0926 19:39:52.734543 (+ 24092us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.734549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.734567 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:39:52.734619 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.734624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.734708 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.734830 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:52.735925 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.737080 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.739087 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.739750 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.739848 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.745236 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:52.745243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:52.745247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:52.745285 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:52.748651 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.749474 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.749479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.749483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.749634 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.750032 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.750037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.750041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.750374 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:52.750488 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.750989 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.751201 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23972,"thread_start_us":24771,"threads_started":1}
I20260926 19:39:52.752588 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.711072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35039 (local address 127.2.164.67:33433)
0926 19:39:52.728202 (+ 17130us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:52.728219 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:52.728330 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:52.734876 (+ 6546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:52.734881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:52.735466 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:52.735743 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.735753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.737296 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.737301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.738908 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.738922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.740162 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.740168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.740348 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.741012 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:52.741032 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:52.745019 (+ 3987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:52.748813 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.748819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.748823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.749345 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.749777 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.749782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.749786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.749919 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.750573 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:52.750578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:52.750728 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 19:39:52.751867 (+ 1139us) client_negotiation.cc:240] Negotiation successful
0926 19:39:52.752147 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17002,"mutex_wait_us":45,"thread_start_us":17187,"threads_started":1}
I20260926 19:39:52.753846 10096 heartbeater.cc:347] Connected to a master server at 127.2.164.126:39835
I20260926 19:39:52.754168 10096 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:52.755023 10096 heartbeater.cc:511] Master 127.2.164.126:39835 requested a full tablet report, sending...
I20260926 19:39:52.756655 8845 ts_manager.cc:194] Re-registered known tserver with Master: 5cf6d073c00548fc93ebea3723743c47 (127.2.164.67:42421)
I20260926 19:39:52.757771 8845 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:37323
I20260926 19:39:52.770678 10097 heartbeater.cc:347] Connected to a master server at 127.2.164.125:35039
I20260926 19:39:52.770798 10097 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:52.771235 10097 heartbeater.cc:511] Master 127.2.164.125:35039 requested a full tablet report, sending...
I20260926 19:39:52.772393 9442 ts_manager.cc:194] Registered new tserver with Master: 5cf6d073c00548fc93ebea3723743c47 (127.2.164.67:42421)
Sep 26 19:39:52 dist-test-slave-mrw4 krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451592, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.124@KRBTEST.COM
I20260926 19:39:52.782704 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.712149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:37681 (local address 127.2.164.124:35195)
0926 19:39:52.733478 (+ 21329us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.733485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.734084 (+ 599us) server_negotiation.cc:407] Connection header received
0926 19:39:52.735454 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.735459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.735538 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.735637 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:52.737079 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.738079 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.745449 (+ 7370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.746082 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.746202 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.775113 (+ 28911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:52.775120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:52.775126 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:52.775170 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:52.779094 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.779832 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.779838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.779842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.779981 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.780411 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.780415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.780419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.780749 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:52.780887 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.782080 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.782325 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21208,"thread_start_us":21336,"threads_started":1}
I20260926 19:39:52.784096 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.711747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:35195 (local address 127.2.164.67:37681)
0926 19:39:52.733967 (+ 22220us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:52.733985 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:52.734115 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:52.736153 (+ 2038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:52.736159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:52.736618 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:52.736898 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.736910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.738787 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.738792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.744264 (+ 5472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.744280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.768245 (+ 23965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.768252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.768454 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.770708 (+ 2254us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:52.770730 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:52.774855 (+ 4125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:52.779248 (+ 4393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.779253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.779256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.779686 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.780128 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:52.780132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:52.780136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:52.780288 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:52.780976 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:52.780980 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:52.781146 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 19:39:52.783337 (+ 2191us) client_negotiation.cc:240] Negotiation successful
0926 19:39:52.783676 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22099,"thread_start_us":21951,"threads_started":1}
I20260926 19:39:52.784914 10095 heartbeater.cc:347] Connected to a master server at 127.2.164.124:35195
I20260926 19:39:52.785043 10095 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:52.785398 10095 heartbeater.cc:511] Master 127.2.164.124:35195 requested a full tablet report, sending...
I20260926 19:39:52.786553 9629 ts_manager.cc:194] Registered new tserver with Master: 5cf6d073c00548fc93ebea3723743c47 (127.2.164.67:42421)
I20260926 19:39:52.796098 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.742753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.2.164.126:39835)
0926 19:39:52.765187 (+ 22434us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.765194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.776729 (+ 11535us) server_negotiation.cc:407] Connection header received
0926 19:39:52.776999 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.777006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.777092 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.777154 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:52.780528 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.781564 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.785848 (+ 4284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.786434 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.786518 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.790138 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:52.790145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:52.790150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:52.790188 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:52.792867 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.793721 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.793726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.793730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.793871 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.794810 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.794816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.794820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.795152 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:52.795377 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.795669 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.795866 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22312,"thread_start_us":33092,"threads_started":1}
I20260926 19:39:52.797709 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.743667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.2.164.125:35039)
0926 19:39:52.751822 (+ 8155us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.751827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.781963 (+ 30136us) server_negotiation.cc:407] Connection header received
0926 19:39:52.782188 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.782194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.782293 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.782358 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:52.785218 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.786278 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.788070 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.788644 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.788741 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.791476 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:52.791483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:52.791487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:52.791522 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:52.794350 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.794989 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.794994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.794997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.795127 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.795488 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.795493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.795497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.796088 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:52.796596 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.797181 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.797392 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8041,"thread_start_us":26268,"threads_started":1}
I20260926 19:39:52.799135 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.742537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.2.164.124:35195)
0926 19:39:52.782936 (+ 40399us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.782942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.782959 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:39:52.783004 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.783009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.783075 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.783167 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:52.787530 (+ 4363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.788431 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.790259 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.790624 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.790723 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.793509 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:52.793515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:52.793520 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:52.793553 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:52.796504 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.797248 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.797252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.797255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.797419 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:52.797898 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:52.797903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:52.797906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:52.798136 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:52.798328 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.798656 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.798850 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40301,"thread_start_us":44808,"threads_started":1}
I20260926 19:39:52.799958 8845 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:39:52.806326 8845 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:52.836597 10126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.823720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:34565 (local address 127.0.0.1:51802)
0926 19:39:52.825229 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:52.825247 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:52.825367 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:52.829561 (+ 4194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:52.829567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:52.829595 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:52.829957 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.829970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.831738 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.831747 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.833967 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.833978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.835872 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.835879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.836032 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.836065 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:39:52.836160 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 19:39:52.836286 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1374,"thread_start_us":6169,"threads_started":1}
I20260926 19:39:52.837371 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.824182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.2.164.66:34565)
0926 19:39:52.828894 (+ 4712us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.828904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.828935 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:39:52.829038 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.829048 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.829365 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.829567 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:52.830162 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.831604 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.834099 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.835475 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.835598 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.835806 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.836920 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.837083 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4553,"thread_start_us":4448,"threads_started":1}
I20260926 19:39:52.837476 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.824148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:45559 (local address 127.0.0.1:43336)
0926 19:39:52.830139 (+ 5991us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:52.830162 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:52.830285 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:52.830853 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:52.830858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:52.830880 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:52.831154 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.831164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.832861 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.832867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.834674 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.834685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.836758 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.836764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.836926 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.836957 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:39:52.837040 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 19:39:52.837146 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5888,"mutex_wait_us":23,"thread_start_us":793,"threads_started":1}
I20260926 19:39:52.837890 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.824607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.2.164.65:45559)
0926 19:39:52.828382 (+ 3775us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.828392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.830270 (+ 1878us) server_negotiation.cc:407] Connection header received
0926 19:39:52.830419 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.830428 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.830681 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.830864 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:52.831358 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.832738 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.834803 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.836346 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.836459 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.836647 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.837102 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.837291 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3616,"thread_start_us":3511,"threads_started":1}
I20260926 19:39:52.840670 9888 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef3914e8e874323b13482e028d31485 (DEFAULT_TABLE table=test-table [id=1cdc444875bd407ea1e37306254867ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:52.840876 9742 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef3914e8e874323b13482e028d31485 (DEFAULT_TABLE table=test-table [id=1cdc444875bd407ea1e37306254867ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:52.841970 9888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef3914e8e874323b13482e028d31485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:52.842031 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef3914e8e874323b13482e028d31485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:52.847361 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.823162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42421 (local address 127.0.0.1:43268)
0926 19:39:52.823693 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:52.823710 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:52.823837 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:52.831358 (+ 7521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:52.831366 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:52.831389 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:52.831919 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.831935 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.834109 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.834114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.836197 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.836209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.838162 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.838167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.846802 (+ 8635us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.846839 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 19:39:52.846928 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 19:39:52.847066 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":6,"thread_start_us":277,"threads_started":1}
I20260926 19:39:52.848075 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.823408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.2.164.67:42421)
0926 19:39:52.830782 (+ 7374us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.830790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.830831 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:39:52.830919 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.830926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.831125 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.831435 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:52.832115 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.833704 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.836379 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.837900 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.838026 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.838215 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.847587 (+ 9372us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.847755 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7220,"thread_start_us":7078,"threads_started":1}
I20260926 19:39:52.851504 10030 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef3914e8e874323b13482e028d31485 (DEFAULT_TABLE table=test-table [id=1cdc444875bd407ea1e37306254867ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:52.852811 10030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef3914e8e874323b13482e028d31485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:52.853722 10129 tablet_bootstrap.cc:492] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91: Bootstrap starting.
I20260926 19:39:52.856638 10129 tablet_bootstrap.cc:654] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:52.858175 10129 log.cc:824] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:52.860749 10129 tablet_bootstrap.cc:492] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91: No bootstrap required, opened a new log
I20260926 19:39:52.860985 10130 tablet_bootstrap.cc:492] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3: Bootstrap starting.
I20260926 19:39:52.861081 10129 ts_tablet_manager.cc:1397] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260926 19:39:52.864306 10130 tablet_bootstrap.cc:654] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:52.865890 10130 log.cc:824] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:52.866725 10132 tablet_bootstrap.cc:492] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: Bootstrap starting.
I20260926 19:39:52.869871 10130 tablet_bootstrap.cc:492] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3: No bootstrap required, opened a new log
I20260926 19:39:52.870126 10130 ts_tablet_manager.cc:1397] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 19:39:52.870709 10132 tablet_bootstrap.cc:654] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:52.871054 10129 raft_consensus.cc:348] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:52.871491 10129 raft_consensus.cc:374] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:52.871578 10129 raft_consensus.cc:729] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b82043831211489bbc3c7f33beb81c91, State: Initialized, Role: FOLLOWER
I20260926 19:39:52.872186 10129 consensus_queue.cc:260] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:52.872553 10132 log.cc:824] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:52.873821 10129 ts_tablet_manager.cc:1428] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 19:39:52.880221 10130 raft_consensus.cc:348] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:52.880506 10130 raft_consensus.cc:374] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:52.880543 10132 tablet_bootstrap.cc:492] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: No bootstrap required, opened a new log
I20260926 19:39:52.880589 10130 raft_consensus.cc:729] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a543749a3e4f2fa93f5de7075a88d3, State: Initialized, Role: FOLLOWER
I20260926 19:39:52.880764 10132 ts_tablet_manager.cc:1397] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260926 19:39:52.881086 10130 consensus_queue.cc:260] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:52.882071 9954 heartbeater.cc:503] Master 127.2.164.126:39835 was elected leader, sending a full tablet report...
I20260926 19:39:52.882727 10130 ts_tablet_manager.cc:1428] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260926 19:39:52.888725 10132 raft_consensus.cc:348] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:52.889020 10132 raft_consensus.cc:374] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:52.889108 10132 raft_consensus.cc:729] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf6d073c00548fc93ebea3723743c47, State: Initialized, Role: FOLLOWER
I20260926 19:39:52.889639 10132 consensus_queue.cc:260] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:52.890499 10096 heartbeater.cc:503] Master 127.2.164.126:39835 was elected leader, sending a full tablet report...
I20260926 19:39:52.891033 10132 ts_tablet_manager.cc:1428] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
W20260926 19:39:52.977483 10098 tablet.cc:2369] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:52.982612 10134 raft_consensus.cc:482] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:52.982914 10134 raft_consensus.cc:504] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:52.984414 10134 leader_election.cc:288] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cf6d073c00548fc93ebea3723743c47 (127.2.164.67:42421), a3a543749a3e4f2fa93f5de7075a88d3 (127.2.164.66:34565)
I20260926 19:39:52.991123 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.984731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:43397 (local address 127.2.164.67:42421)
0926 19:39:52.984892 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.984896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.985939 (+ 1043us) server_negotiation.cc:407] Connection header received
0926 19:39:52.986113 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.986119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.986195 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.986274 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:52.986836 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.987587 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.989503 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.990201 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.990318 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.990386 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.990773 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.990895 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 19:39:52.991135 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.985487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:48629 (local address 127.2.164.66:34565)
0926 19:39:52.985632 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:52.985636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:52.986163 (+ 527us) server_negotiation.cc:407] Connection header received
0926 19:39:52.986316 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:52.986321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:52.986396 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:52.986487 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:52.986968 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.987633 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.989495 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.990253 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.990408 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.990478 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:52.990772 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 19:39:52.990901 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 19:39:52.991288 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.985423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:34565 (local address 127.2.164.65:48629)
0926 19:39:52.986054 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:52.986069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:52.986186 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:52.986505 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:52.986508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:52.986525 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:52.986822 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.986832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.987761 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.987764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.989350 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.989361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.990463 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.990467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.990613 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.990654 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 19:39:52.990728 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 19:39:52.990886 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":113,"thread_start_us":364,"threads_started":1}
I20260926 19:39:52.991326 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:52.984640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42421 (local address 127.2.164.65:43397)
0926 19:39:52.985813 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:52.985836 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:52.985988 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:52.986303 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:52.986307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:52.986333 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:52.986692 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.986702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.987719 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.987725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.989360 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:52.989371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:52.990399 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:52.990405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:52.990612 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:52.990651 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 19:39:52.990728 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 19:39:52.990897 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":82,"thread_start_us":288,"threads_started":1}
I20260926 19:39:52.992146 9908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef3914e8e874323b13482e028d31485" candidate_uuid: "b82043831211489bbc3c7f33beb81c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" is_pre_election: true
I20260926 19:39:52.992146 10050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef3914e8e874323b13482e028d31485" candidate_uuid: "b82043831211489bbc3c7f33beb81c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cf6d073c00548fc93ebea3723743c47" is_pre_election: true
I20260926 19:39:52.992583 10050 raft_consensus.cc:2454] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b82043831211489bbc3c7f33beb81c91 in term 0.
I20260926 19:39:52.992583 9908 raft_consensus.cc:2454] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b82043831211489bbc3c7f33beb81c91 in term 0.
I20260926 19:39:52.993135 9697 leader_election.cc:302] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3a543749a3e4f2fa93f5de7075a88d3, b82043831211489bbc3c7f33beb81c91; no voters:
I20260926 19:39:52.993494 10134 raft_consensus.cc:2781] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:52.993614 10134 raft_consensus.cc:482] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:52.993702 10134 raft_consensus.cc:3037] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:52.996229 10134 raft_consensus.cc:504] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:52.996739 10134 leader_election.cc:288] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [CANDIDATE]: Term 1 election: Requested vote from peers 5cf6d073c00548fc93ebea3723743c47 (127.2.164.67:42421), a3a543749a3e4f2fa93f5de7075a88d3 (127.2.164.66:34565)
I20260926 19:39:52.997148 10050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef3914e8e874323b13482e028d31485" candidate_uuid: "b82043831211489bbc3c7f33beb81c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cf6d073c00548fc93ebea3723743c47"
I20260926 19:39:52.997241 9908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef3914e8e874323b13482e028d31485" candidate_uuid: "b82043831211489bbc3c7f33beb81c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a543749a3e4f2fa93f5de7075a88d3"
I20260926 19:39:52.997391 10050 raft_consensus.cc:3037] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:52.997418 9908 raft_consensus.cc:3037] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:53.000384 9908 raft_consensus.cc:2454] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b82043831211489bbc3c7f33beb81c91 in term 1.
I20260926 19:39:53.000386 10050 raft_consensus.cc:2454] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b82043831211489bbc3c7f33beb81c91 in term 1.
I20260926 19:39:53.000727 9697 leader_election.cc:302] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cf6d073c00548fc93ebea3723743c47, b82043831211489bbc3c7f33beb81c91; no voters:
I20260926 19:39:53.000968 10134 raft_consensus.cc:2781] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:53.001410 10134 raft_consensus.cc:686] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 1 LEADER]: Becoming Leader. State: Replica: b82043831211489bbc3c7f33beb81c91, State: Running, Role: LEADER
I20260926 19:39:53.001901 10134 consensus_queue.cc:237] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:53.006572 8845 catalog_manager.cc:5495] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 reported cstate change: term changed from 0 to 1, leader changed from <none> to b82043831211489bbc3c7f33beb81c91 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "b82043831211489bbc3c7f33beb81c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } health_report { overall_health: HEALTHY } } }
I20260926 19:39:53.024219 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:53.019106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.2.164.65:45559)
0926 19:39:53.019298 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:53.019302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:53.019318 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:39:53.019444 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:53.019447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:53.019514 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:53.019625 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:53.019864 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:53.020916 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:53.022184 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:53.022576 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:53.022706 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:53.023152 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:53.023666 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:53.023791 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:53.023879 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 19:39:53.023952 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 19:39:53.031600 10050 raft_consensus.cc:1261] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47 [term 1 FOLLOWER]: Refusing update from remote peer b82043831211489bbc3c7f33beb81c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:53.031601 9908 raft_consensus.cc:1261] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3 [term 1 FOLLOWER]: Refusing update from remote peer b82043831211489bbc3c7f33beb81c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:53.032514 10140 consensus_queue.cc:1035] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:53.032836 10134 consensus_queue.cc:1035] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:53.045787 10143 mvcc.cc:205] Tried to move back new op lower bound from 7333689725048512512 to 7333689724939157504. Current Snapshot: MvccSnapshot[applied={T|T < 7333689725048512512}]
I20260926 19:39:53.060472 8844 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:53.060760 8844 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:53.067203 8844 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1547d4ec04a04ec58ebfae894e02340f: Sending DeleteTablet for 3 replicas of tablet 8ef3914e8e874323b13482e028d31485
I20260926 19:39:53.068178 10030 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef3914e8e874323b13482e028d31485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:53 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:43268
I20260926 19:39:53.068416 9888 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef3914e8e874323b13482e028d31485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:53 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:51802
I20260926 19:39:53.068845 9742 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef3914e8e874323b13482e028d31485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:53 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:43336
I20260926 19:39:53.069301 10154 tablet_replica.cc:330] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3: stopping tablet replica
I20260926 19:39:53.069765 10154 raft_consensus.cc:2229] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:53.070350 10154 raft_consensus.cc:2258] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:53.070778 10155 tablet_replica.cc:330] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91: stopping tablet replica
I20260926 19:39:53.071257 10155 raft_consensus.cc:2229] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:39:53.071827 10155 raft_consensus.cc:2258] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:53.073326 10154 ts_tablet_manager.cc:1905] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:53.074039 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 8813
I20260926 19:39:53.074936 8875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:53.081039 10153 tablet_replica.cc:330] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: stopping tablet replica
I20260926 19:39:53.081483 10153 raft_consensus.cc:2229] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:53.081950 10153 raft_consensus.cc:2258] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:53.082068 10155 ts_tablet_manager.cc:1905] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:53.083925 10153 ts_tablet_manager.cc:1905] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:53.087131 10154 ts_tablet_manager.cc:1918] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:53.090169 10153 ts_tablet_manager.cc:1918] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:53.090339 10153 log.cc:1196] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/8ef3914e8e874323b13482e028d31485
I20260926 19:39:53.090454 10154 log.cc:1196] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/8ef3914e8e874323b13482e028d31485
I20260926 19:39:53.090826 10153 ts_tablet_manager.cc:1939] T 8ef3914e8e874323b13482e028d31485 P 5cf6d073c00548fc93ebea3723743c47: Deleting consensus metadata
I20260926 19:39:53.091377 10154 ts_tablet_manager.cc:1939] T 8ef3914e8e874323b13482e028d31485 P a3a543749a3e4f2fa93f5de7075a88d3: Deleting consensus metadata
I20260926 19:39:53.094040 10155 ts_tablet_manager.cc:1918] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:53.094194 10155 log.cc:1196] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/8ef3914e8e874323b13482e028d31485
I20260926 19:39:53.094740 10155 ts_tablet_manager.cc:1939] T 8ef3914e8e874323b13482e028d31485 P b82043831211489bbc3c7f33beb81c91: Deleting consensus metadata
I20260926 19:39:53.099578 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:53.071832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59678 (local address 127.2.164.125:35039)
0926 19:39:53.071983 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:53.071989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:53.072007 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:39:53.075296 (+ 3289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:53.075301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:53.075381 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:53.075453 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:53.082155 (+ 6702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:53.083224 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:53.087339 (+ 4115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:53.087943 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:53.088036 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:53.093971 (+ 5935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:53.093977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:53.093982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:53.094023 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:53.096921 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:53.097806 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:53.097812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:53.097816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:53.097933 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:53.098261 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:53.098266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:53.098270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:53.098523 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:53.098748 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:53.099054 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 19:39:53.099253 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:39:53.111881 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:53.072852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.2.164.124:35195)
0926 19:39:53.072989 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:53.072994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:53.099919 (+ 26925us) server_negotiation.cc:407] Connection header received
0926 19:39:53.100192 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:53.100198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:53.100288 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:53.100356 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:53.101909 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:53.102602 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:53.103809 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:53.104227 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:53.104347 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:53.106740 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:53.106745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:53.106749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:53.106780 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:53.109562 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:53.110101 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:53.110105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:53.110107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:53.110238 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:53.110584 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:53.110587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:53.110590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:53.110844 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:53.111067 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:53.111345 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 19:39:53.111544 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 19:39:53.248387 8831 catalog_manager.cc:4841] TS 5cf6d073c00548fc93ebea3723743c47 (127.2.164.67:42421): tablet 8ef3914e8e874323b13482e028d31485 (table test-table [id=1cdc444875bd407ea1e37306254867ca]) successfully deleted
W20260926 19:39:53.248678 8831 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.2.164.67:42421, user_credentials={real_user=slave}} blocked reactor thread for 156194us
I20260926 19:39:53.248773 8832 catalog_manager.cc:4841] TS b82043831211489bbc3c7f33beb81c91 (127.2.164.65:45559): tablet 8ef3914e8e874323b13482e028d31485 (table test-table [id=1cdc444875bd407ea1e37306254867ca]) successfully deleted
I20260926 19:39:53.249511 8831 catalog_manager.cc:4841] TS a3a543749a3e4f2fa93f5de7075a88d3 (127.2.164.66:34565): tablet 8ef3914e8e874323b13482e028d31485 (table test-table [id=1cdc444875bd407ea1e37306254867ca]) successfully deleted
I20260926 19:39:53.249740 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 8813
I20260926 19:39:53.302853 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:53.285267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.2.164.124:35195)
0926 19:39:53.285475 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:53.285481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:53.286091 (+ 610us) server_negotiation.cc:407] Connection header received
0926 19:39:53.286380 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:53.286386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:53.286463 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:53.286528 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:53.288803 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:53.289868 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:53.291664 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:53.292266 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:53.292358 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:53.297299 (+ 4941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:53.297305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:53.297311 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:53.297353 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:53.300079 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:53.300865 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:53.300871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:53.300875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:53.301020 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:53.301393 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:53.301398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:53.301402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:53.301735 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:53.301963 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:53.302260 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 19:39:53.302464 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:39:53.311372 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:53.286230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.2.164.125:35039)
0926 19:39:53.286423 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:53.286429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:53.296679 (+ 10250us) server_negotiation.cc:407] Connection header received
0926 19:39:53.296958 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:53.296964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:53.297044 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:53.297151 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:53.298885 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:53.299929 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:53.302016 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:53.302669 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:53.302770 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:53.305871 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:53.305877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:53.305881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:53.305918 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:53.309023 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:53.309638 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:53.309643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:53.309646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:53.309785 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:53.310069 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:53.310073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:53.310076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:53.310418 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:53.310647 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:53.310819 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 19:39:53.311024 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:39:54.052693 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.051963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.65:41457)
0926 19:39:54.052395 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.052512 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":232,"threads_started":1}
I20260926 19:39:54.052760 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.052073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.66:41089)
0926 19:39:54.052473 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.052572 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":204,"threads_started":1}
W20260926 19:39:54.053182 9808 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:39835 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
W20260926 19:39:54.053378 9954 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:39835 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
I20260926 19:39:54.057566 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.056796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.67:42063)
0926 19:39:54.057272 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.057397 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"spinlock_wait_cycles":896,"thread_start_us":294,"threads_started":1}
W20260926 19:39:54.057938 10096 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:39835 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
I20260926 19:39:54.567656 10168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1547d4ec04a04ec58ebfae894e02340f)
I20260926 19:39:54.567907 10169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1547d4ec04a04ec58ebfae894e02340f)
I20260926 19:39:54.567940 10168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } }
I20260926 19:39:54.568145 10169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } }
I20260926 19:39:54.569561 10169 leader_election.cc:288] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1547d4ec04a04ec58ebfae894e02340f (127.2.164.126:39835), 7e76dcb9e13f4dc2a2bdd4eda6ebe25d (127.2.164.125:35039)
I20260926 19:39:54.570753 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.569655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56886)
0926 19:39:54.570175 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.570455 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"threads_started":1}
W20260926 19:39:54.571743 9613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
I20260926 19:39:54.575825 10168 leader_election.cc:288] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1547d4ec04a04ec58ebfae894e02340f (127.2.164.126:39835), 96d207b9c12c4630b062682fbe003611 (127.2.164.124:35195)
I20260926 19:39:54.581817 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.569740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56898)
0926 19:39:54.581280 (+ 11540us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.581499 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11403,"thread_start_us":11257,"threads_started":1}
W20260926 19:39:54.582120 9426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
I20260926 19:39:54.585434 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.584467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56914)
0926 19:39:54.584990 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.585225 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393}
W20260926 19:39:54.585744 9613 leader_election.cc:334] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1547d4ec04a04ec58ebfae894e02340f (127.2.164.126:39835): Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
I20260926 19:39:54.586827 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.569792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35039 (local address 127.0.0.1:59694)
0926 19:39:54.571053 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.571072 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:54.571203 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:54.579320 (+ 8117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:54.579326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:54.579357 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:54.579616 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.579629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.582064 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.582070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.584266 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.584284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.586110 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.586116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.586293 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.586326 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:39:54.586409 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 19:39:54.586542 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":10906,"threads_started":1}
I20260926 19:39:54.587334 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.570070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.2.164.125:35039)
0926 19:39:54.578353 (+ 8283us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:54.578359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:54.578377 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:39:54.578448 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:54.578453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:54.578521 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:54.579774 (+ 1253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:54.580049 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.581360 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.584520 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.585918 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.586040 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.586143 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:54.586544 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 19:39:54.586801 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":8147,"thread_start_us":7985,"threads_started":1}
I20260926 19:39:54.587422 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.587091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56920)
0926 19:39:54.587215 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.587297 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 19:39:54.587622 9426 leader_election.cc:334] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1547d4ec04a04ec58ebfae894e02340f (127.2.164.126:39835): Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
I20260926 19:39:54.587827 9451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96d207b9c12c4630b062682fbe003611" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" is_pre_election: true
I20260926 19:39:54.588173 9451 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96d207b9c12c4630b062682fbe003611 in term 1.
I20260926 19:39:54.588625 9616 leader_election.cc:302] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e76dcb9e13f4dc2a2bdd4eda6ebe25d, 96d207b9c12c4630b062682fbe003611; no voters: 1547d4ec04a04ec58ebfae894e02340f
I20260926 19:39:54.588982 10169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 19:39:54.589118 10169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1547d4ec04a04ec58ebfae894e02340f)
I20260926 19:39:54.589238 10169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:39:54.590579 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.578522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:35195 (local address 127.0.0.1:55842)
0926 19:39:54.581579 (+ 3057us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.581600 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:54.581717 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:54.584012 (+ 2295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:54.584016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:54.584040 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:54.584464 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.584477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.586208 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.586213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.588510 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.588521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.590065 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.590069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.590186 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.590209 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:39:54.590282 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 19:39:54.590370 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2946,"thread_start_us":2806,"threads_started":1}
I20260926 19:39:54.590786 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.583093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.2.164.124:35195)
0926 19:39:54.583739 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:54.583744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:54.583764 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:39:54.583814 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:54.583819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:54.583875 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:54.584609 (+ 734us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:54.584815 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.586070 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.588656 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.589904 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.590039 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.590144 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:54.590318 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 19:39:54.590444 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":420,"threads_started":1}
I20260926 19:39:54.591322 9639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "96d207b9c12c4630b062682fbe003611" is_pre_election: true
I20260926 19:39:54.591869 10169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } }
I20260926 19:39:54.592237 9639 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e76dcb9e13f4dc2a2bdd4eda6ebe25d in current term 2: Already voted for candidate 96d207b9c12c4630b062682fbe003611 in this term.
I20260926 19:39:54.592448 10169 leader_election.cc:288] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [CANDIDATE]: Term 2 election: Requested vote from peers 1547d4ec04a04ec58ebfae894e02340f (127.2.164.126:39835), 7e76dcb9e13f4dc2a2bdd4eda6ebe25d (127.2.164.125:35039)
I20260926 19:39:54.592866 9451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96d207b9c12c4630b062682fbe003611" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d"
I20260926 19:39:54.592963 9428 leader_election.cc:302] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e76dcb9e13f4dc2a2bdd4eda6ebe25d; no voters: 1547d4ec04a04ec58ebfae894e02340f, 96d207b9c12c4630b062682fbe003611
I20260926 19:39:54.593057 9451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:39:54.593518 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.592508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56936)
0926 19:39:54.593295 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.593385 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":688}
I20260926 19:39:54.594303 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.594025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56938)
0926 19:39:54.594148 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.594218 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 19:39:54.594450 9613 leader_election.cc:334] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1547d4ec04a04ec58ebfae894e02340f (127.2.164.126:39835): Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
I20260926 19:39:54.595202 9451 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96d207b9c12c4630b062682fbe003611 in term 2.
I20260926 19:39:54.595355 10168 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:39:54.595549 9616 leader_election.cc:302] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e76dcb9e13f4dc2a2bdd4eda6ebe25d, 96d207b9c12c4630b062682fbe003611; no voters: 1547d4ec04a04ec58ebfae894e02340f
I20260926 19:39:54.595793 10169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 19:39:54.596225 10169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [term 2 LEADER]: Becoming Leader. State: Replica: 96d207b9c12c4630b062682fbe003611, State: Running, Role: LEADER
I20260926 19:39:54.596606 10169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } }
I20260926 19:39:54.598328 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96d207b9c12c4630b062682fbe003611. Latest consensus state: current_term: 2 leader_uuid: "96d207b9c12c4630b062682fbe003611" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:54.598704 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:54.599254 10180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:54.602892 10180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:54.603668 10180 catalog_manager.cc:1259] Loaded cluster ID: cf759cf0b7094eb98ac9c84b7965c53e
I20260926 19:39:54.603734 10180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:54.604789 10180 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:54.605592 10180 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: Loaded TSK: 0
I20260926 19:39:54.606360 10180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:39:54.641762 9629 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:39:54.648494 9629 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:54.652136 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.651793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56960)
0926 19:39:54.651941 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.652010 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 19:39:54.652275 9451 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [term 2 FOLLOWER]: Refusing update from remote peer 96d207b9c12c4630b062682fbe003611: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 19:39:54.652940 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.652625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56976)
0926 19:39:54.652753 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.652826 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 19:39:54.652987 10178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 19:39:54.653267 9613 consensus_peers.cc:487] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 -> Peer 1547d4ec04a04ec58ebfae894e02340f (127.2.164.126:39835): Couldn't send request to peer 1547d4ec04a04ec58ebfae894e02340f. Status: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 19:39:54.655514 10168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96d207b9c12c4630b062682fbe003611. Latest consensus state: current_term: 2 leader_uuid: "96d207b9c12c4630b062682fbe003611" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:54.655746 10168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:39:54.656949 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "96d207b9c12c4630b062682fbe003611" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:54.657181 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:54.657574 10168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "96d207b9c12c4630b062682fbe003611" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1547d4ec04a04ec58ebfae894e02340f" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 39835 } } peers { permanent_uuid: "7e76dcb9e13f4dc2a2bdd4eda6ebe25d" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35039 } attrs { promote: false } } peers { permanent_uuid: "96d207b9c12c4630b062682fbe003611" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 35195 } attrs { promote: false } } }
I20260926 19:39:54.657826 10168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e76dcb9e13f4dc2a2bdd4eda6ebe25d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:39:54.673780 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.665022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:34565 (local address 127.0.0.1:51816)
0926 19:39:54.665439 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.665456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:54.665554 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:54.666696 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:54.666702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:54.666730 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:54.667103 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.667115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.668896 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.668903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.671138 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.671154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.672922 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.672929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.673127 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.673184 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 19:39:54.673252 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 19:39:54.673417 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316}
I20260926 19:39:54.674974 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.665864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.2.164.66:34565)
0926 19:39:54.666348 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:54.666355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:54.666374 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:39:54.666451 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:54.666456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:54.666534 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:54.667303 (+ 769us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:54.667470 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.668743 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.671426 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.672670 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.674358 (+ 1688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.674471 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:54.674523 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 19:39:54.674661 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":395,"threads_started":1}
I20260926 19:39:54.681874 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.664916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:45559 (local address 127.0.0.1:43360)
0926 19:39:54.665062 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.665083 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:54.665200 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:54.677760 (+ 12560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:54.677764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:54.677796 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:54.677923 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.677930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.679023 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.679028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.680174 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.680184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.681216 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.681222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.681389 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.681411 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:39:54.681484 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 19:39:54.681607 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 19:39:54.681910 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.665021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.2.164.65:45559)
0926 19:39:54.677516 (+ 12495us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:54.677520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:54.677536 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:39:54.677587 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:54.677591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:54.677644 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:54.677730 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:54.678062 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.678890 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.680319 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.681048 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.681156 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.681269 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:54.681526 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 19:39:54.681661 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12395,"thread_start_us":12310,"threads_started":1}
I20260926 19:39:54.682739 9742 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea25d1557b04ba4857e75efff580926 (DEFAULT_TABLE table=test-table [id=ff0592f0685345aa972f9e74bdd01dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:54.683136 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea25d1557b04ba4857e75efff580926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:54.684670 9888 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea25d1557b04ba4857e75efff580926 (DEFAULT_TABLE table=test-table [id=ff0592f0685345aa972f9e74bdd01dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:54.685139 9888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea25d1557b04ba4857e75efff580926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:54.688076 10191 tablet_bootstrap.cc:492] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91: Bootstrap starting.
I20260926 19:39:54.689848 10191 tablet_bootstrap.cc:654] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:54.690986 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.666452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42421 (local address 127.0.0.1:43284)
0926 19:39:54.673987 (+ 7535us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.674005 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:54.674097 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:54.686018 (+ 11921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:54.686023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:54.686050 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:54.686194 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.686203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.687479 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.687484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.688969 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.688980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.690348 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.690352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.690475 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.690497 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:39:54.690571 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 19:39:54.690698 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7445,"thread_start_us":17363,"threads_started":1}
I20260926 19:39:54.691042 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.666144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.2.164.67:42421)
0926 19:39:54.685697 (+ 19553us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:54.685703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:54.685722 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:39:54.685785 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:54.685790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:54.685873 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:54.685973 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:54.686332 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.687337 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.689112 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.690072 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.690179 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.690281 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:54.690628 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 19:39:54.690758 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19436,"thread_start_us":19930,"threads_started":1}
I20260926 19:39:54.691838 10030 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea25d1557b04ba4857e75efff580926 (DEFAULT_TABLE table=test-table [id=ff0592f0685345aa972f9e74bdd01dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:54.691929 10193 tablet_bootstrap.cc:492] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3: Bootstrap starting.
I20260926 19:39:54.692224 10030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea25d1557b04ba4857e75efff580926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:54.692425 10191 tablet_bootstrap.cc:492] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91: No bootstrap required, opened a new log
I20260926 19:39:54.692521 10191 ts_tablet_manager.cc:1397] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 19:39:54.693192 10191 raft_consensus.cc:348] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:54.693444 10191 raft_consensus.cc:374] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:54.693534 10191 raft_consensus.cc:729] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b82043831211489bbc3c7f33beb81c91, State: Initialized, Role: FOLLOWER
I20260926 19:39:54.693578 10193 tablet_bootstrap.cc:654] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:54.693738 10191 consensus_queue.cc:260] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:54.694451 10191 ts_tablet_manager.cc:1428] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 19:39:54.696002 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.695197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.65:51327)
0926 19:39:54.695653 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.695803 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"threads_started":1}
I20260926 19:39:54.698241 9807 heartbeater.cc:503] Master 127.2.164.124:35195 was elected leader, sending a full tablet report...
I20260926 19:39:54.700421 10197 tablet_bootstrap.cc:492] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47: Bootstrap starting.
I20260926 19:39:54.702099 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.699983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.65:53395)
0926 19:39:54.700212 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.700293 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260926 19:39:54.702109 10197 tablet_bootstrap.cc:654] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:54.703672 10193 tablet_bootstrap.cc:492] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3: No bootstrap required, opened a new log
I20260926 19:39:54.703809 10193 ts_tablet_manager.cc:1397] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260926 19:39:54.704038 10197 tablet_bootstrap.cc:492] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47: No bootstrap required, opened a new log
I20260926 19:39:54.704166 10197 ts_tablet_manager.cc:1397] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 19:39:54.704484 10193 raft_consensus.cc:348] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:54.704686 10193 raft_consensus.cc:374] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:54.704763 10193 raft_consensus.cc:729] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a543749a3e4f2fa93f5de7075a88d3, State: Initialized, Role: FOLLOWER
I20260926 19:39:54.704805 10197 raft_consensus.cc:348] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:54.705000 10197 raft_consensus.cc:374] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:54.704977 10193 consensus_queue.cc:260] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:54.705089 10197 raft_consensus.cc:729] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf6d073c00548fc93ebea3723743c47, State: Initialized, Role: FOLLOWER
I20260926 19:39:54.705332 10197 consensus_queue.cc:260] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:54.705689 10193 ts_tablet_manager.cc:1428] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:54.707798 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.706170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.66:50627)
0926 19:39:54.707372 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.707546 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1080,"thread_start_us":346,"threads_started":1}
I20260926 19:39:54.709506 9953 heartbeater.cc:503] Master 127.2.164.124:35195 was elected leader, sending a full tablet report...
I20260926 19:39:54.713541 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.708402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.66:51005)
0926 19:39:54.713355 (+ 4953us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.713423 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4879,"threads_started":1}
I20260926 19:39:54.717653 10197 ts_tablet_manager.cc:1428] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260926 19:39:54.719388 10095 heartbeater.cc:503] Master 127.2.164.124:35195 was elected leader, sending a full tablet report...
I20260926 19:39:54.719597 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.717877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.67:40335)
0926 19:39:54.719206 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.719365 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1193,"thread_start_us":251,"threads_started":1}
I20260926 19:39:54.720458 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.720161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.67:50459)
0926 19:39:54.720296 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.720364 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 19:39:54.733227 10200 raft_consensus.cc:482] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:54.733390 10200 raft_consensus.cc:504] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:54.734827 10200 leader_election.cc:288] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3a543749a3e4f2fa93f5de7075a88d3 (127.2.164.66:34565), b82043831211489bbc3c7f33beb81c91 (127.2.164.65:45559)
I20260926 19:39:54.740890 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.735035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:35575 (local address 127.2.164.66:34565)
0926 19:39:54.735218 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:54.735224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:54.735307 (+ 83us) server_negotiation.cc:407] Connection header received
0926 19:39:54.735599 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:54.735604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:54.735685 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:54.735744 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:54.736330 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.737009 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.738850 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.739987 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.740110 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.740171 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:54.740519 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 19:39:54.740645 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:39:54.740898 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.735372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:44709 (local address 127.2.164.65:45559)
0926 19:39:54.735567 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:54.735573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:54.735592 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:39:54.735815 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:54.735820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:54.735894 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:54.735980 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:54.736440 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.737085 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.738742 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.739866 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.740024 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.740096 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:54.740531 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 19:39:54.740660 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:39:54.741192 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.734956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:34565 (local address 127.2.164.67:35575)
0926 19:39:54.735177 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.735195 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:54.735434 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:54.735909 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:54.735914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:54.735937 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:54.736206 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.736217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.737133 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.737139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.738631 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.738644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.740223 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.740228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.740364 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.740407 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 19:39:54.740479 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 19:39:54.740690 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":82}
I20260926 19:39:54.741222 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.735010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:45559 (local address 127.2.164.67:44709)
0926 19:39:54.735540 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.735555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:54.735639 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:54.736013 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:54.736017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:54.736040 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:54.736307 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.736317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.737227 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.737234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.738614 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:54.738626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.740090 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.740097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:54.740353 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.740403 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 19:39:54.740480 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 19:39:54.740657 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":124,"thread_start_us":594,"threads_started":1}
I20260926 19:39:54.741791 9908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea25d1557b04ba4857e75efff580926" candidate_uuid: "5cf6d073c00548fc93ebea3723743c47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" is_pre_election: true
I20260926 19:39:54.741871 9762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea25d1557b04ba4857e75efff580926" candidate_uuid: "5cf6d073c00548fc93ebea3723743c47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b82043831211489bbc3c7f33beb81c91" is_pre_election: true
I20260926 19:39:54.742002 9908 raft_consensus.cc:2454] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cf6d073c00548fc93ebea3723743c47 in term 0.
I20260926 19:39:54.742231 9762 raft_consensus.cc:2454] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cf6d073c00548fc93ebea3723743c47 in term 0.
I20260926 19:39:54.742426 9985 leader_election.cc:302] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cf6d073c00548fc93ebea3723743c47, a3a543749a3e4f2fa93f5de7075a88d3; no voters:
I20260926 19:39:54.742780 10200 raft_consensus.cc:2781] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:54.742897 10200 raft_consensus.cc:482] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:54.742983 10200 raft_consensus.cc:3037] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:54.744927 10200 raft_consensus.cc:504] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:54.745491 10200 leader_election.cc:288] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [CANDIDATE]: Term 1 election: Requested vote from peers a3a543749a3e4f2fa93f5de7075a88d3 (127.2.164.66:34565), b82043831211489bbc3c7f33beb81c91 (127.2.164.65:45559)
I20260926 19:39:54.745811 9908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea25d1557b04ba4857e75efff580926" candidate_uuid: "5cf6d073c00548fc93ebea3723743c47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a543749a3e4f2fa93f5de7075a88d3"
I20260926 19:39:54.745890 9762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea25d1557b04ba4857e75efff580926" candidate_uuid: "5cf6d073c00548fc93ebea3723743c47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b82043831211489bbc3c7f33beb81c91"
I20260926 19:39:54.745965 9908 raft_consensus.cc:3037] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:54.746048 9762 raft_consensus.cc:3037] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:54.748042 9908 raft_consensus.cc:2454] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cf6d073c00548fc93ebea3723743c47 in term 1.
I20260926 19:39:54.748222 9762 raft_consensus.cc:2454] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cf6d073c00548fc93ebea3723743c47 in term 1.
I20260926 19:39:54.748409 9985 leader_election.cc:302] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cf6d073c00548fc93ebea3723743c47, a3a543749a3e4f2fa93f5de7075a88d3; no voters:
I20260926 19:39:54.748642 10200 raft_consensus.cc:2781] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:54.749065 10200 raft_consensus.cc:686] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 1 LEADER]: Becoming Leader. State: Replica: 5cf6d073c00548fc93ebea3723743c47, State: Running, Role: LEADER
I20260926 19:39:54.749449 10200 consensus_queue.cc:237] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } } peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } }
I20260926 19:39:54.750130 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.749744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.67:49563)
0926 19:39:54.749899 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.749998 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 19:39:54.752736 9629 catalog_manager.cc:5495] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cf6d073c00548fc93ebea3723743c47 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "5cf6d073c00548fc93ebea3723743c47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cf6d073c00548fc93ebea3723743c47" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 } health_report { overall_health: UNKNOWN } } }
I20260926 19:39:54.774125 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.769662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.2.164.67:42421)
0926 19:39:54.769802 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:54.769806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:54.769822 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:39:54.769989 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:54.769992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:54.770059 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:54.770168 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:54.770428 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.771091 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.772380 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.772763 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.772850 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.773140 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:54.773662 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:54.773760 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:54.773887 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 19:39:54.773943 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 19:39:54.777560 9908 raft_consensus.cc:1261] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3 [term 1 FOLLOWER]: Refusing update from remote peer 5cf6d073c00548fc93ebea3723743c47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:54.777854 9762 raft_consensus.cc:1261] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91 [term 1 FOLLOWER]: Refusing update from remote peer 5cf6d073c00548fc93ebea3723743c47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:54.778201 10205 consensus_queue.cc:1035] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3a543749a3e4f2fa93f5de7075a88d3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:54.778542 10200 consensus_queue.cc:1035] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b82043831211489bbc3c7f33beb81c91" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:54.781056 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.780397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.66:60283)
0926 19:39:54.780646 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.780835 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 19:39:54.781127 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.780652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.65:49937)
0926 19:39:54.780854 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.780932 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 19:39:54.785293 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.784781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.66:52093)
0926 19:39:54.785045 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.785123 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260926 19:39:54.785655 10208 mvcc.cc:205] Tried to move back new op lower bound from 7333689732202979328 to 7333689732097642496. Current Snapshot: MvccSnapshot[applied={T|T < 7333689732202979328}]
I20260926 19:39:54.785914 10209 mvcc.cc:205] Tried to move back new op lower bound from 7333689732202979328 to 7333689732097642496. Current Snapshot: MvccSnapshot[applied={T|T < 7333689732202979328}]
I20260926 19:39:54.790019 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.789599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.67:37347)
0926 19:39:54.789777 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.789872 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 19:39:54.797511 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.796923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.67:38765)
0926 19:39:54.797267 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.797351 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252}
I20260926 19:39:54.798492 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.798041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.2.164.65:57375)
0926 19:39:54.798271 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:54.798358 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260926 19:39:54.801302 9629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:54.801551 9629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:39:54.808593 9742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea25d1557b04ba4857e75efff580926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:54 UTC) from {username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:43360
I20260926 19:39:54.808744 9629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96d207b9c12c4630b062682fbe003611: Sending DeleteTablet for 3 replicas of tablet 2ea25d1557b04ba4857e75efff580926
I20260926 19:39:54.809697 9888 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea25d1557b04ba4857e75efff580926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:54 UTC) from {username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:51816
I20260926 19:39:54.809722 10030 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea25d1557b04ba4857e75efff580926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:54 UTC) from {username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:43284
I20260926 19:39:54.813005 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 9606
I20260926 19:39:54.814857 10220 tablet_replica.cc:330] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47: stopping tablet replica
I20260926 19:39:54.815021 10220 raft_consensus.cc:2229] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:39:54.815441 10220 raft_consensus.cc:2258] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:54.816040 9741 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea25d1557b04ba4857e75efff580926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:39:54 UTC) from {username='oryx', principal='oryx/127.2.164.124@KRBTEST.COM'} at 127.0.0.1:43360
I20260926 19:39:54.816524 10219 tablet_replica.cc:330] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3: stopping tablet replica
I20260926 19:39:54.816718 10219 raft_consensus.cc:2229] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:54.817000 10219 raft_consensus.cc:2258] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:54.817494 10218 tablet_replica.cc:330] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91: stopping tablet replica
I20260926 19:39:54.817653 10218 raft_consensus.cc:2229] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:39:54.817893 10218 raft_consensus.cc:2258] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:39:54.818200 10219 ts_tablet_manager.cc:1905] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:54.818524 10220 ts_tablet_manager.cc:1905] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:54.820015 10218 ts_tablet_manager.cc:1905] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:39:54.824481 10219 ts_tablet_manager.cc:1918] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:54.824600 10219 log.cc:1196] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/2ea25d1557b04ba4857e75efff580926
I20260926 19:39:54.825032 10219 ts_tablet_manager.cc:1939] T 2ea25d1557b04ba4857e75efff580926 P a3a543749a3e4f2fa93f5de7075a88d3: Deleting consensus metadata
I20260926 19:39:54.826056 10218 ts_tablet_manager.cc:1918] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:54.826073 9615 catalog_manager.cc:4841] TS a3a543749a3e4f2fa93f5de7075a88d3 (127.2.164.66:34565): tablet 2ea25d1557b04ba4857e75efff580926 (table test-table [id=ff0592f0685345aa972f9e74bdd01dca]) successfully deleted
I20260926 19:39:54.826170 10218 log.cc:1196] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/2ea25d1557b04ba4857e75efff580926
I20260926 19:39:54.826506 10218 ts_tablet_manager.cc:1939] T 2ea25d1557b04ba4857e75efff580926 P b82043831211489bbc3c7f33beb81c91: Deleting consensus metadata
W20260926 19:39:54.827704 9616 catalog_manager.cc:4808] TS b82043831211489bbc3c7f33beb81c91 (127.2.164.65:45559): delete failed for tablet 2ea25d1557b04ba4857e75efff580926 because the tablet was not found. No further retry: Not found: Tablet not found: 2ea25d1557b04ba4857e75efff580926
I20260926 19:39:54.829279 10220 ts_tablet_manager.cc:1918] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:39:54.829417 10220 log.cc:1196] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/2ea25d1557b04ba4857e75efff580926
I20260926 19:39:54.829844 10220 ts_tablet_manager.cc:1939] T 2ea25d1557b04ba4857e75efff580926 P 5cf6d073c00548fc93ebea3723743c47: Deleting consensus metadata
I20260926 19:39:54.830813 9615 catalog_manager.cc:4841] TS 5cf6d073c00548fc93ebea3723743c47 (127.2.164.67:42421): tablet 2ea25d1557b04ba4857e75efff580926 (table test-table [id=ff0592f0685345aa972f9e74bdd01dca]) successfully deleted
I20260926 19:39:54.842262 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:54.814103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.2.164.65:45559)
0926 19:39:54.814247 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:54.814252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:54.815979 (+ 1727us) server_negotiation.cc:407] Connection header received
0926 19:39:54.816151 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:54.816156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:54.816233 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:54.816336 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:54.818013 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.819065 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.824658 (+ 5593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:54.825267 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:54.825342 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:54.836954 (+ 11612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:54.836967 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:54.836982 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:54.837030 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:54.839641 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:54.840435 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:54.840441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:54.840444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:54.840555 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:54.840912 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:54.840915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:54.840917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:54.841302 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:54.841513 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:54.841849 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 19:39:54.841999 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:39:54.842862 9802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:55.018955 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 9606
I20260926 19:39:55.044651 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 9819
I20260926 19:39:55.057281 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:55.045439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.2.164.66:34565)
0926 19:39:55.045643 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:55.045648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:55.045667 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:39:55.045808 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:55.045813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:55.045886 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:55.045965 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:55.047166 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:55.048008 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:55.049334 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:55.049717 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:55.049848 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:55.052207 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:55.052220 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:55.052235 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:55.052277 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:55.054789 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:55.055393 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:55.055400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:55.055402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:55.055517 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:55.055884 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:55.055889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:55.055892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:55.056211 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:55.056467 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:55.056756 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 19:39:55.056959 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 19:39:55.057886 9948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:55.208571 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 9819
I20260926 19:39:55.229665 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 9966
I20260926 19:39:55.242909 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:55.230524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.2.164.67:42421)
0926 19:39:55.230702 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:55.230707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:55.230726 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:39:55.230860 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:55.230865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:55.230949 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:55.231034 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:55.232317 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:55.233077 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:55.234287 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:55.234689 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:55.234800 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:55.237212 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:55.237228 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:55.237252 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:55.237288 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:55.240206 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:55.240819 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:55.240827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:55.240830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:55.240992 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:55.241365 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:55.241370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:55.241373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:55.241826 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:55.242152 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:55.242381 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 19:39:55.242590 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:39:55.243551 10090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:55.254129 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:55.253479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56982)
0926 19:39:55.253861 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:55.253976 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":194,"threads_started":1}
I20260926 19:39:55.255326 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:55.255019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:39835 (local address 127.0.0.1:56998)
0926 19:39:55.255146 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:55.255217 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 19:39:55.368074 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 9966
I20260926 19:39:55.384624 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 9306
I20260926 19:39:55.385686 9472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:55.499459 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 9306
I20260926 19:39:55.515959 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 9406
I20260926 19:39:55.516937 9659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:39:55.649793 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 9406
2026-09-26T19:39:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8932 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:39:55 dist-test-slave-mrw4 krb5kdc[10231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:39:55 dist-test-slave-mrw4 krb5kdc[10231](info): set up 2 sockets
Sep 26 19:39:55 dist-test-slave-mrw4 krb5kdc[10231](info): commencing operation
krb5kdc: starting...
W20260926 19:39:56.637702 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:39:56 dist-test-slave-mrw4 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:39:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.126.keytab.
Entry for principal oryx/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.126.keytab.
I20260926 19:39:56.819730 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:37235
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.126.keytab
--principal=oryx/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:37235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:57.117259 10244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:57.117553 10244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:57.117681 10244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:57.124555 10244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:39:57.124650 10244 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 19:39:57.124712 10244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:57.124794 10244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:57.124854 10244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:39:57.124933 10244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:39:57.134318 10244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:37235
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.126.keytab
--principal=oryx/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:37235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:57.135461 10244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:57.136680 10244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:57.147228 10249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:57.147214 10250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:57.147253 10252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:57.147964 10244 server_base.cc:1034] running on GCE node
I20260926 19:39:57.148677 10244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:57.149961 10244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:57.151161 10244 hybrid_clock.cc:648] HybridClock initialized: now 1790451597151088 us; error 55 us; skew 500 ppm
I20260926 19:39:57.151446 10244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:57 dist-test-slave-mrw4 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451597, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:57.154296 10244 init.cc:377] Logged in from keytab as oryx/127.2.164.126@KRBTEST.COM (short username oryx)
I20260926 19:39:57.155546 10244 webserver.cc:460] Webserver started at http://127.2.164.126:37063/ using document root <none> and password file <none>
I20260926 19:39:57.156186 10244 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:57.156271 10244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:57.156558 10244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:57.159281 10244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "2c75419b28ee4f71963f77741ce55882"
format_stamp: "Formatted at 2026-09-26 19:39:57 on dist-test-slave-mrw4"
server_key: "8572961f221ec10db2beb29ba5a7a734"
server_key_iv: "bab22ba9bf3ef1236220df2ac885b9a3"
server_key_version: "encryptionkey@0"
I20260926 19:39:57.160223 10244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "2c75419b28ee4f71963f77741ce55882"
format_stamp: "Formatted at 2026-09-26 19:39:57 on dist-test-slave-mrw4"
server_key: "8572961f221ec10db2beb29ba5a7a734"
server_key_iv: "bab22ba9bf3ef1236220df2ac885b9a3"
server_key_version: "encryptionkey@0"
I20260926 19:39:57.165035 10244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 19:39:57.168259 10259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:57.169571 10244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:39:57.169708 10244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "2c75419b28ee4f71963f77741ce55882"
format_stamp: "Formatted at 2026-09-26 19:39:57 on dist-test-slave-mrw4"
server_key: "8572961f221ec10db2beb29ba5a7a734"
server_key_iv: "bab22ba9bf3ef1236220df2ac885b9a3"
server_key_version: "encryptionkey@0"
I20260926 19:39:57.169895 10244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:57.192624 10244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:57.196951 10244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:57.197225 10244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:57.216976 10244 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:37235
I20260926 19:39:57.216998 10320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:37235 every 8 connection(s)
I20260926 19:39:57.218334 10244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:39:57.222020 10321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:57.225575 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 10244
I20260926 19:39:57.225747 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:39:57.226359 2705 external_mini_cluster.cc:1428] Setting key af58bc350834eb27989498b18f8d8d1e
I20260926 19:39:57.234347 10321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882: Bootstrap starting.
I20260926 19:39:57.237870 10321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:57.239087 10321 log.cc:824] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:39:57 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:39:57.242002 10321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882: No bootstrap required, opened a new log
I20260926 19:39:57.249344 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:57.228222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.2.164.126:37235)
0926 19:39:57.228945 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:57.228956 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:57.228995 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:39:57.230560 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:57.230581 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:57.230982 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:57.231304 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:57.232935 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:57.234295 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:57.236016 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:57.236668 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:57.236793 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:57.241446 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:57.241453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:57.241473 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:57.241508 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:57.245092 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:57.245738 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:57.245753 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:57.245762 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:57.245960 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:57.246315 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:57.246320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:57.246324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:57.246963 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:57.247247 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:57.247628 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 19:39:57.248039 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":330,"threads_started":1}
I20260926 19:39:57.251703 10321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c75419b28ee4f71963f77741ce55882" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37235 } }
I20260926 19:39:57.252099 10321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:57.252188 10321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c75419b28ee4f71963f77741ce55882, State: Initialized, Role: FOLLOWER
I20260926 19:39:57.252645 10321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c75419b28ee4f71963f77741ce55882" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37235 } }
I20260926 19:39:57.252831 10321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:39:57.252918 10321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:39:57.253029 10321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:57.255133 10321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c75419b28ee4f71963f77741ce55882" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37235 } }
I20260926 19:39:57.255549 10321 leader_election.cc:302] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c75419b28ee4f71963f77741ce55882; no voters:
I20260926 19:39:57.255980 10321 leader_election.cc:288] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:39:57.256100 10326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:57.256690 10326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [term 1 LEADER]: Becoming Leader. State: Replica: 2c75419b28ee4f71963f77741ce55882, State: Running, Role: LEADER
I20260926 19:39:57.257215 10326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c75419b28ee4f71963f77741ce55882" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37235 } }
I20260926 19:39:57.257730 10321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:39:57.259698 10327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c75419b28ee4f71963f77741ce55882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c75419b28ee4f71963f77741ce55882" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37235 } } }
I20260926 19:39:57.260144 10327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:57.260219 10328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c75419b28ee4f71963f77741ce55882. Latest consensus state: current_term: 1 leader_uuid: "2c75419b28ee4f71963f77741ce55882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c75419b28ee4f71963f77741ce55882" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37235 } } }
I20260926 19:39:57.260463 10328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882 [sys.catalog]: This master's current role is: LEADER
I20260926 19:39:57.261757 10331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:39:57.266903 10331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:39:57.275803 10331 catalog_manager.cc:1347] Generated new cluster ID: fd4c26d656734cc1b54890ffdae94738
I20260926 19:39:57.275902 10331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:39:57.313186 10331 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:39:57.314566 10331 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:39:57.321866 10331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882: Generated new TSK 0
I20260926 19:39:57.322741 10331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab.
I20260926 19:39:57.365594 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:37235
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab
--principal=oryx/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:57.670020 10347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:57.670293 10347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:57.670388 10347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:57.677574 10347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:57.677675 10347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:57.677793 10347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:39:57.677893 10347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:39:57.687173 10347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab
--principal=oryx/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:37235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:57.688401 10347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:57.689679 10347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:57.701844 10355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:57.701807 10353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:57.701877 10352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:57.702428 10347 server_base.cc:1034] running on GCE node
I20260926 19:39:57.702980 10347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:57.703796 10347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:57.704989 10347 hybrid_clock.cc:648] HybridClock initialized: now 1790451597704936 us; error 35 us; skew 500 ppm
I20260926 19:39:57.705271 10347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:57 dist-test-slave-mrw4 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451597, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:57.708042 10347 init.cc:377] Logged in from keytab as oryx/127.2.164.65@KRBTEST.COM (short username oryx)
I20260926 19:39:57.709358 10347 webserver.cc:460] Webserver started at http://127.2.164.65:45301/ using document root <none> and password file <none>
I20260926 19:39:57.709965 10347 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:57.710045 10347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:57.710283 10347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:57.712874 10347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc"
format_stamp: "Formatted at 2026-09-26 19:39:57 on dist-test-slave-mrw4"
server_key: "bc9650c8a1286d6a87bf495d939f18bc"
server_key_iv: "a4dad7835c0330bc1d104f98494bcc1b"
server_key_version: "encryptionkey@0"
I20260926 19:39:57.713610 10347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc"
format_stamp: "Formatted at 2026-09-26 19:39:57 on dist-test-slave-mrw4"
server_key: "bc9650c8a1286d6a87bf495d939f18bc"
server_key_iv: "a4dad7835c0330bc1d104f98494bcc1b"
server_key_version: "encryptionkey@0"
I20260926 19:39:57.718127 10347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:39:57.721138 10362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:57.722393 10347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 19:39:57.722527 10347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc"
format_stamp: "Formatted at 2026-09-26 19:39:57 on dist-test-slave-mrw4"
server_key: "bc9650c8a1286d6a87bf495d939f18bc"
server_key_iv: "a4dad7835c0330bc1d104f98494bcc1b"
server_key_version: "encryptionkey@0"
I20260926 19:39:57.722708 10347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:57.735517 10347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:57.738945 10347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:57.739161 10347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:57.741580 10347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:57.741672 10347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:57.741820 10347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:57.741937 10347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 26 19:39:57 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451597, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:39:57.761276 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:57.742020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:43593 (local address 127.2.164.126:37235)
0926 19:39:57.742176 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:57.742182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:57.743541 (+ 1359us) server_negotiation.cc:407] Connection header received
0926 19:39:57.745532 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:57.745538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:57.745615 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:57.745701 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:57.747709 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:57.748854 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:57.751274 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:57.751811 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:57.751926 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:57.755638 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:57.755643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:57.755646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:57.755678 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:57.758175 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:57.759037 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:57.759065 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:57.759069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:57.759232 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:57.759592 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:57.759597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:57.759600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:57.759939 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:57.760080 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:57.760657 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 19:39:57.760879 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 19:39:57.762272 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:57.742581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37235 (local address 127.2.164.65:43593)
0926 19:39:57.743379 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:57.743429 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:57.745271 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:57.745954 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:57.745962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:57.746664 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:57.747461 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:57.747480 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:57.749033 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:57.749042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:57.751137 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:57.751155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:57.752054 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:57.752066 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:57.752261 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:57.752966 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:57.752997 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:57.755437 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:57.758299 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:57.758308 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:57.758326 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:57.758886 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:57.759334 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:57.759337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:57.759340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:57.759499 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:57.760057 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:57.760062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:57.760343 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 19:39:57.760549 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 19:39:57.760922 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":318,"threads_started":1}
I20260926 19:39:57.780202 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:57.767407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:40657 (local address 127.2.164.126:37235)
0926 19:39:57.767548 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:57.767553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:57.768328 (+ 775us) server_negotiation.cc:407] Connection header received
0926 19:39:57.768527 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:57.768533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:57.768606 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:57.768673 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:57.770013 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:57.770906 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:57.772695 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:57.773316 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:57.773441 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:57.775249 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:57.775254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:57.775257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:57.775283 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:57.777510 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:57.778267 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:57.778272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:57.778276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:57.778422 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:57.778831 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:57.778836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:57.778839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:57.779166 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:57.779290 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:57.779664 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 19:39:57.779866 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 19:39:57.780267 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:57.767480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37235 (local address 127.2.164.65:40657)
0926 19:39:57.768175 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:57.768243 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:57.768381 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:57.768847 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:57.768850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:57.769400 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:57.769830 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:57.769846 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:57.771075 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:57.771078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:57.772566 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:57.772584 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:57.773520 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:57.773527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:57.773763 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:57.774364 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:57.774380 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:57.775094 (+ 714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:57.777623 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:57.777628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:57.777632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:57.778157 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:57.778535 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:57.778541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:57.778544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:57.778712 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:57.779280 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:57.779288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:57.779449 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 19:39:57.779562 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 19:39:57.779843 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":345,"threads_started":1}
I20260926 19:39:57.793959 10347 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:41579
I20260926 19:39:57.793980 10481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:41579 every 8 connection(s)
I20260926 19:39:57.795047 10347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:39:57.796713 10482 heartbeater.cc:347] Connected to a master server at 127.2.164.126:37235
I20260926 19:39:57.796890 10482 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:57.797482 10482 heartbeater.cc:511] Master 127.2.164.126:37235 requested a full tablet report, sending...
I20260926 19:39:57.799206 10276 ts_manager.cc:194] Registered new tserver with Master: c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579)
I20260926 19:39:57.800536 10276 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:43593
I20260926 19:39:57.804951 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 10347
I20260926 19:39:57.805110 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:39:57.805724 2705 external_mini_cluster.cc:1428] Setting key 96bc7ae28b024740ad956377b9b53296
WARNING: no policy specified for oryx/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab.
I20260926 19:39:57.843761 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:37235
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab
--principal=oryx/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:58.140827 10486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:58.141083 10486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:58.141222 10486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:58.148123 10486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:58.148231 10486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:58.148339 10486 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:39:58.148437 10486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:39:58.157545 10486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab
--principal=oryx/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:37235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:58.158699 10486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:58.159961 10486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:58.171751 10491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:58.171795 10492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:58.171782 10494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:58.172502 10486 server_base.cc:1034] running on GCE node
I20260926 19:39:58.173060 10486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:58.173892 10486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:58.175096 10486 hybrid_clock.cc:648] HybridClock initialized: now 1790451598175015 us; error 55 us; skew 500 ppm
I20260926 19:39:58.175347 10486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:58 dist-test-slave-mrw4 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451598, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:58.178108 10486 init.cc:377] Logged in from keytab as oryx/127.2.164.66@KRBTEST.COM (short username oryx)
I20260926 19:39:58.179333 10486 webserver.cc:460] Webserver started at http://127.2.164.66:45737/ using document root <none> and password file <none>
I20260926 19:39:58.179956 10486 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:58.180047 10486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:58.180317 10486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:58.183019 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "170d8a6a16b642aba38cd6fb1762151e"
format_stamp: "Formatted at 2026-09-26 19:39:58 on dist-test-slave-mrw4"
server_key: "3a2831a65b688e3a36d9c98f0780d53e"
server_key_iv: "ad2ce0010bc18c35004faceefe6aafda"
server_key_version: "encryptionkey@0"
I20260926 19:39:58.183732 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "170d8a6a16b642aba38cd6fb1762151e"
format_stamp: "Formatted at 2026-09-26 19:39:58 on dist-test-slave-mrw4"
server_key: "3a2831a65b688e3a36d9c98f0780d53e"
server_key_iv: "ad2ce0010bc18c35004faceefe6aafda"
server_key_version: "encryptionkey@0"
I20260926 19:39:58.188340 10486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 19:39:58.191385 10501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:58.192535 10486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:39:58.192670 10486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "170d8a6a16b642aba38cd6fb1762151e"
format_stamp: "Formatted at 2026-09-26 19:39:58 on dist-test-slave-mrw4"
server_key: "3a2831a65b688e3a36d9c98f0780d53e"
server_key_iv: "ad2ce0010bc18c35004faceefe6aafda"
server_key_version: "encryptionkey@0"
I20260926 19:39:58.192843 10486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:58.205940 10486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:58.209390 10486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:58.209611 10486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:58.211926 10486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:58.212000 10486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:58.212128 10486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:58.212185 10486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:39:58 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451598, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:39:58.237028 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:58.212546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:50135 (local address 127.2.164.126:37235)
0926 19:39:58.212709 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:58.212715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:58.214067 (+ 1352us) server_negotiation.cc:407] Connection header received
0926 19:39:58.216710 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:58.216717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:58.216799 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:58.216868 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:58.219281 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.220436 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.223008 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.223674 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.223791 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:58.228554 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:58.228561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:58.228566 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:58.228602 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:58.232035 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:58.233035 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:58.233042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:58.233046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:58.233214 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:58.233789 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:58.233795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:58.233798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:58.234124 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:58.236397 (+ 2273us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:58.236531 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 19:39:58.236743 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:39:58.237093 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:58.213042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37235 (local address 127.2.164.66:50135)
0926 19:39:58.213878 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:58.213940 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:58.216373 (+ 2433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:58.217415 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:58.217429 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:58.218211 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:58.219068 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:58.219096 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.220626 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.220631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:58.222848 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:58.222864 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.224064 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.224071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:58.224314 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:58.225063 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:58.225092 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:58.228294 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:58.232202 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:58.232211 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:58.232225 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:58.232885 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:58.233368 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:58.233373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:58.233381 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:58.233648 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:58.234280 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:58.234287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:58.234717 (+ 430us) client_negotiation.cc:769] Sending connection context
0926 19:39:58.234993 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 19:39:58.235547 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":347,"threads_started":1}
I20260926 19:39:58.252903 10486 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:32951
I20260926 19:39:58.252938 10620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:32951 every 8 connection(s)
I20260926 19:39:58.254112 10486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:39:58.256143 10621 heartbeater.cc:347] Connected to a master server at 127.2.164.126:37235
I20260926 19:39:58.256346 10621 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:58.256883 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:58.243561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:60039 (local address 127.2.164.126:37235)
0926 19:39:58.243718 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:58.243721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:58.244637 (+ 916us) server_negotiation.cc:407] Connection header received
0926 19:39:58.244875 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:58.244881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:58.244957 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:58.245053 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:58.246162 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.247310 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.249059 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.249546 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.249669 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:58.251441 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:58.251447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:58.251451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:58.251484 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:58.254097 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:58.254839 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:58.254844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:58.254847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:58.254973 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:58.255450 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:58.255455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:58.255458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:58.255844 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:58.255974 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:58.256354 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 19:39:58.256550 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 19:39:58.257004 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:58.243630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37235 (local address 127.2.164.66:60039)
0926 19:39:58.244541 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:58.244559 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:58.244701 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:58.245093 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:58.245096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:58.245609 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:58.246025 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:58.246043 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.247427 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.247430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:58.248883 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:58.248901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.249730 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.249736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:58.249910 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:58.250526 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:58.250542 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:58.251236 (+ 694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:58.254218 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:58.254223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:58.254227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:58.254696 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:58.255117 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:58.255122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:58.255125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:58.255347 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:58.255948 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:58.255953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:58.256134 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 19:39:58.256255 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 19:39:58.256550 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":589,"threads_started":1}
I20260926 19:39:58.257021 10621 heartbeater.cc:511] Master 127.2.164.126:37235 requested a full tablet report, sending...
I20260926 19:39:58.258456 10275 ts_manager.cc:194] Registered new tserver with Master: 170d8a6a16b642aba38cd6fb1762151e (127.2.164.66:32951)
I20260926 19:39:58.259577 10275 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:50135
I20260926 19:39:58.260245 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 10486
I20260926 19:39:58.260385 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:39:58.260864 2705 external_mini_cluster.cc:1428] Setting key 10021b8c7142a4101cf3e3a52daaff14
WARNING: no policy specified for oryx/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab.
I20260926 19:39:58.300320 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:37235
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab
--principal=oryx/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:39:58.599933 10625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:39:58.600171 10625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:39:58.600337 10625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:39:58.607182 10625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:39:58.607281 10625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:39:58.607398 10625 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:39:58.607496 10625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:39:58.616598 10625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab
--principal=oryx/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.164.126:37235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:39:58.617765 10625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:39:58.618983 10625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:39:58.630642 10630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:58.630589 10631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:39:58.631263 10633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:39:58.631260 10625 server_base.cc:1034] running on GCE node
I20260926 19:39:58.631886 10625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:39:58.632682 10625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:39:58.633872 10625 hybrid_clock.cc:648] HybridClock initialized: now 1790451598633813 us; error 39 us; skew 500 ppm
I20260926 19:39:58.634124 10625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:39:58 dist-test-slave-mrw4 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451598, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:39:58.636991 10625 init.cc:377] Logged in from keytab as oryx/127.2.164.67@KRBTEST.COM (short username oryx)
I20260926 19:39:58.638242 10625 webserver.cc:460] Webserver started at http://127.2.164.67:38261/ using document root <none> and password file <none>
I20260926 19:39:58.638831 10625 fs_manager.cc:359] Metadata directory not provided
I20260926 19:39:58.638909 10625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:39:58.639209 10625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:39:58.641928 10625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "1426574353c44bfab197662b80c87417"
format_stamp: "Formatted at 2026-09-26 19:39:58 on dist-test-slave-mrw4"
server_key: "c988b63076f99c83eb13699aa34494d7"
server_key_iv: "e16d3604872a68676c6cc5d6c465adf5"
server_key_version: "encryptionkey@0"
I20260926 19:39:58.642623 10625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "1426574353c44bfab197662b80c87417"
format_stamp: "Formatted at 2026-09-26 19:39:58 on dist-test-slave-mrw4"
server_key: "c988b63076f99c83eb13699aa34494d7"
server_key_iv: "e16d3604872a68676c6cc5d6c465adf5"
server_key_version: "encryptionkey@0"
I20260926 19:39:58.647171 10625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:39:58.650272 10640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:58.651433 10625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:39:58.651559 10625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "1426574353c44bfab197662b80c87417"
format_stamp: "Formatted at 2026-09-26 19:39:58 on dist-test-slave-mrw4"
server_key: "c988b63076f99c83eb13699aa34494d7"
server_key_iv: "e16d3604872a68676c6cc5d6c465adf5"
server_key_version: "encryptionkey@0"
I20260926 19:39:58.651731 10625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:39:58.681478 10625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:39:58.684962 10625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:39:58.685205 10625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:39:58.687692 10625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:39:58.687769 10625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:39:58.687901 10625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:39:58.687954 10625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:39:58 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451598, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:39:58.709625 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:58.688248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:60497 (local address 127.2.164.126:37235)
0926 19:39:58.688438 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:58.688445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:58.689852 (+ 1407us) server_negotiation.cc:407] Connection header received
0926 19:39:58.691987 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:58.691994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:58.692074 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:58.692149 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:58.694374 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.695464 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.697746 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.698358 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.698477 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:58.702861 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:58.702866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:58.702869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:58.702900 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:58.706213 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:58.707090 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:58.707095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:58.707098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:58.707220 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:58.707762 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:58.707766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:58.707768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:58.708079 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:58.708239 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:58.709194 (+ 955us) server_negotiation.cc:299] Negotiation successful
0926 19:39:58.709378 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:39:58.711319 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:58.688863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37235 (local address 127.2.164.67:60497)
0926 19:39:58.689679 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:58.689729 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:58.691644 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:58.692626 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:58.692645 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:58.693390 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:58.694178 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:58.694205 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.695614 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.695619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:58.697606 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:58.697626 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.698593 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.698600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:58.698792 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:58.699486 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:58.699520 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:58.702621 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:58.706354 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:58.706367 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:58.706389 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:58.706928 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:58.707322 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:58.707329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:58.707332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:58.707605 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:58.708246 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:58.708256 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:58.708758 (+ 502us) client_negotiation.cc:769] Sending connection context
0926 19:39:58.709091 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 19:39:58.709704 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":337,"threads_started":1}
I20260926 19:39:58.730432 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:58.717285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:47765 (local address 127.2.164.126:37235)
0926 19:39:58.717452 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:58.717458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:58.718565 (+ 1107us) server_negotiation.cc:407] Connection header received
0926 19:39:58.718800 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:58.718806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:58.718869 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:58.718943 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:58.719765 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.720718 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.722766 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.723427 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.723558 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:58.725450 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:58.725456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:58.725461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:58.725498 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:58.728175 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:58.728750 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:58.728755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:58.728759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:58.728893 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:58.729218 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:58.729223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:58.729227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:58.729462 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:58.729596 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:58.730042 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 19:39:58.730184 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 19:39:58.730681 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:58.717769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37235 (local address 127.2.164.67:47765)
0926 19:39:58.718456 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:58.718475 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:58.718622 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:58.718992 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:58.718996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:58.719383 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:39:58.719643 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:58.719651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.720861 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.720868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:58.722565 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:58.722582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.723584 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.723588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:58.723708 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:58.724281 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:39:58.724301 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:39:58.725298 (+ 997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:39:58.728288 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:58.728292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:58.728297 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:58.728643 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:58.728985 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:39:58.728990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:39:58.728994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:39:58.729104 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:39:58.729568 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:39:58.729574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:39:58.729796 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 19:39:58.729929 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 19:39:58.730216 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":401,"threads_started":1}
I20260926 19:39:58.731961 10625 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:35139
I20260926 19:39:58.731985 10759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:35139 every 8 connection(s)
I20260926 19:39:58.733397 10625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:39:58.735163 10760 heartbeater.cc:347] Connected to a master server at 127.2.164.126:37235
I20260926 19:39:58.735337 10760 heartbeater.cc:464] Registering TS with master...
I20260926 19:39:58.735858 10760 heartbeater.cc:511] Master 127.2.164.126:37235 requested a full tablet report, sending...
I20260926 19:39:58.736963 10275 ts_manager.cc:194] Registered new tserver with Master: 1426574353c44bfab197662b80c87417 (127.2.164.67:35139)
I20260926 19:39:58.737550 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 10625
I20260926 19:39:58.737711 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:39:58.737982 10275 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:60497
I20260926 19:39:58.738212 2705 external_mini_cluster.cc:1428] Setting key e3a29c1a5cd3b6a9c13943b0896ebefd
I20260926 19:39:58.740847 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:39:58 dist-test-slave-mrw4 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451598, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 19:39:58 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:39:58 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:39:58.802945 10482 heartbeater.cc:503] Master 127.2.164.126:37235 was elected leader, sending a full tablet report...
I20260926 19:39:59.262195 10621 heartbeater.cc:503] Master 127.2.164.126:37235 was elected leader, sending a full tablet report...
I20260926 19:39:59.740559 10760 heartbeater.cc:503] Master 127.2.164.126:37235 was elected leader, sending a full tablet report...
W20260926 19:39:59.750872 10768 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.971s user 0.000s sys 0.012s
W20260926 19:39:59.751802 10324 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:39:58.774591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.2.164.126:37235)
0926 19:39:58.774789 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:58.774795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:58.775044 (+ 249us) server_negotiation.cc:407] Connection header received
0926 19:39:58.775190 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:58.775194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:58.775257 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:58.775356 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:58.776484 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.777232 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.778347 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:58.778747 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:58.778835 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.751606 (+972771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59902: BlockingRecv error: recv got EOF from 127.0.0.1:59902 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 19:39:59 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451598, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:39:59.767367 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.755184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.2.164.126:37235)
0926 19:39:59.755331 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:59.755336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:59.755687 (+ 351us) server_negotiation.cc:407] Connection header received
0926 19:39:59.755821 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:59.755825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:59.755916 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:59.756005 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:39:59.757142 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.757894 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.759109 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.759542 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.759632 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.762967 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:39:59.762972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:39:59.762975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:39:59.763008 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:39:59.765350 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:59.765900 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:59.765904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:59.765906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:59.766017 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:39:59.766311 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:39:59.766314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:39:59.766316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:39:59.766614 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:39:59.766791 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:59.766989 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 19:39:59.767135 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 19:39:59.770974 10275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:39:59.777436 10275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:39:59.805738 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.787694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41579 (local address 127.0.0.1:48678)
0926 19:39:59.788410 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:59.788446 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:59.788594 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:59.798538 (+ 9944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:59.798546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:59.798581 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:59.798970 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:59.798982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.800861 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.800866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:59.803130 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:59.803143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.804808 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.804813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:59.804977 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.805026 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 19:39:59.805150 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 19:39:59.805347 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":48,"thread_start_us":298,"threads_started":1}
I20260926 19:39:59.806280 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.788816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.2.164.66:32951)
0926 19:39:59.798320 (+ 9504us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:59.798328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:59.798354 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:39:59.798444 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:59.798451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:59.798694 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:59.798872 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:59.799572 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.800950 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.803209 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.804630 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.804766 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.804963 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:59.805822 (+ 859us) server_negotiation.cc:299] Negotiation successful
0926 19:39:59.805983 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4632,"thread_start_us":4502,"threads_started":1}
I20260926 19:39:59.806485 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.787912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.2.164.65:41579)
0926 19:39:59.797924 (+ 10012us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:59.797932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:59.797961 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:39:59.798053 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:59.798060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:59.798304 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:59.799095 (+ 791us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:59.799280 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.800739 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.803253 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.804630 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.804766 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.804968 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:59.806028 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0926 19:39:59.806204 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9878,"thread_start_us":9806,"threads_started":1}
I20260926 19:39:59.806694 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.788227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32951 (local address 127.0.0.1:57174)
0926 19:39:59.789272 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:59.789299 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:59.789410 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:59.798867 (+ 9457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:59.798871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:59.798898 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:59.799378 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:59.799391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.801067 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.801073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:59.803073 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:59.803085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.805126 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.805132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:59.805322 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.805354 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 19:39:59.805444 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 19:39:59.805565 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":69,"spinlock_wait_cycles":1792,"thread_start_us":9160,"threads_started":1}
I20260926 19:39:59.807664 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.788776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:35139 (local address 127.0.0.1:43978)
0926 19:39:59.797531 (+ 8755us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:59.797547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:59.797654 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:59.798980 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:59.798984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:59.799017 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:59.799387 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:59.799395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.800603 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.800608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:59.803132 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:59.803143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.806924 (+ 3781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.806931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:59.807090 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.807119 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 19:39:59.807199 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 19:39:59.807308 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8647,"mutex_wait_us":128,"thread_start_us":629,"threads_started":1}
I20260926 19:39:59.807953 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.789244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.2.164.67:35139)
0926 19:39:59.798245 (+ 9001us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:59.798252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:59.798276 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:39:59.798359 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:59.798365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:59.798572 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:59.798756 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:59.799543 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.800479 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.805050 (+ 4571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.806444 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.806593 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.806788 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:59.807539 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 19:39:59.807688 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8867,"thread_start_us":8746,"threads_started":1}
I20260926 19:39:59.810143 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 892927b1cf504214b967bf7d1469ad28 (DEFAULT_TABLE table=test-table [id=4112bc9097f94342ae63b5035958c2d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:59.810159 10417 tablet_service.cc:1467] Processing CreateTablet for tablet 892927b1cf504214b967bf7d1469ad28 (DEFAULT_TABLE table=test-table [id=4112bc9097f94342ae63b5035958c2d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:59.811265 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892927b1cf504214b967bf7d1469ad28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:59.811456 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892927b1cf504214b967bf7d1469ad28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:59.811167 10690 tablet_service.cc:1467] Processing CreateTablet for tablet 892927b1cf504214b967bf7d1469ad28 (DEFAULT_TABLE table=test-table [id=4112bc9097f94342ae63b5035958c2d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:39:59.812273 10690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892927b1cf504214b967bf7d1469ad28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:39:59.821851 10785 tablet_bootstrap.cc:492] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e: Bootstrap starting.
I20260926 19:39:59.824597 10785 tablet_bootstrap.cc:654] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:59.825348 10786 tablet_bootstrap.cc:492] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Bootstrap starting.
I20260926 19:39:59.825848 10785 log.cc:824] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:59.826690 10787 tablet_bootstrap.cc:492] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: Bootstrap starting.
I20260926 19:39:59.828755 10786 tablet_bootstrap.cc:654] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:59.829921 10787 tablet_bootstrap.cc:654] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: Neither blocks nor log segments found. Creating new log.
I20260926 19:39:59.830026 10786 log.cc:824] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:59.830123 10785 tablet_bootstrap.cc:492] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e: No bootstrap required, opened a new log
I20260926 19:39:59.830403 10785 ts_tablet_manager.cc:1397] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 19:39:59.830926 10787 log.cc:824] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: Log is configured to *not* fsync() on all Append() calls
I20260926 19:39:59.832726 10786 tablet_bootstrap.cc:492] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: No bootstrap required, opened a new log
I20260926 19:39:59.832962 10786 ts_tablet_manager.cc:1397] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 19:39:59.832986 10787 tablet_bootstrap.cc:492] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: No bootstrap required, opened a new log
I20260926 19:39:59.833210 10787 ts_tablet_manager.cc:1397] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 19:39:59.839687 10787 raft_consensus.cc:348] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } }
I20260926 19:39:59.840118 10787 raft_consensus.cc:374] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:59.840231 10787 raft_consensus.cc:729] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1426574353c44bfab197662b80c87417, State: Initialized, Role: FOLLOWER
I20260926 19:39:59.840149 10785 raft_consensus.cc:348] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } }
I20260926 19:39:59.840520 10785 raft_consensus.cc:374] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:59.840615 10785 raft_consensus.cc:729] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170d8a6a16b642aba38cd6fb1762151e, State: Initialized, Role: FOLLOWER
I20260926 19:39:59.840857 10787 consensus_queue.cc:260] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } }
I20260926 19:39:59.841188 10785 consensus_queue.cc:260] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } }
I20260926 19:39:59.842555 10786 raft_consensus.cc:348] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } }
I20260926 19:39:59.842818 10785 ts_tablet_manager.cc:1428] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260926 19:39:59.842988 10786 raft_consensus.cc:374] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:39:59.843132 10786 raft_consensus.cc:729] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dadcd87d4a4d5ea8cc8797b9ea6bcc, State: Initialized, Role: FOLLOWER
I20260926 19:39:59.843755 10786 consensus_queue.cc:260] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } }
I20260926 19:39:59.844198 10787 ts_tablet_manager.cc:1428] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20260926 19:39:59.845597 10786 ts_tablet_manager.cc:1428] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 19:39:59.921446 10792 raft_consensus.cc:482] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:39:59.921726 10792 raft_consensus.cc:504] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } }
I20260926 19:39:59.923254 10792 leader_election.cc:288] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579), 1426574353c44bfab197662b80c87417 (127.2.164.67:35139)
I20260926 19:39:59.929782 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.923626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:53021 (local address 127.2.164.65:41579)
0926 19:39:59.923820 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:59.923826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:59.924265 (+ 439us) server_negotiation.cc:407] Connection header received
0926 19:39:59.924486 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:59.924492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:59.924578 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:59.924634 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:59.925345 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.926223 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.928131 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.928927 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.929061 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.929132 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:59.929447 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 19:39:59.929567 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:39:59.929800 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.924374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:58511 (local address 127.2.164.67:35139)
0926 19:39:59.924560 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:59.924565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:59.924831 (+ 266us) server_negotiation.cc:407] Connection header received
0926 19:39:59.924958 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:59.924962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:59.925042 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:59.925138 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:39:59.925760 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.926548 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.928236 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.928968 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.929111 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.929210 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:59.929468 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 19:39:59.929584 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:39:59.929904 10794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.923591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41579 (local address 127.2.164.66:53021)
0926 19:39:59.924152 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:59.924172 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:59.924318 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:59.924788 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:59.924792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:59.924820 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:59.925209 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:59.925223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.926371 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.926376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:59.927962 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:59.927987 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.929146 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.929152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:59.929307 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.929330 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:39:59.929402 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 19:39:59.929518 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":299,"threads_started":1}
I20260926 19:39:59.930071 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.924225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:35139 (local address 127.2.164.66:58511)
0926 19:39:59.924734 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 19:39:59.924750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:39:59.924847 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:39:59.925241 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:39:59.925245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:39:59.925265 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:39:59.925603 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:59.925615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.926683 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.926687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:59.928096 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:39:59.928111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.929202 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.929209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:39:59.929328 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.929350 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:39:59.929431 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 19:39:59.929555 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":254,"threads_started":1}
I20260926 19:39:59.930825 10710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892927b1cf504214b967bf7d1469ad28" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1426574353c44bfab197662b80c87417" is_pre_election: true
I20260926 19:39:59.930907 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892927b1cf504214b967bf7d1469ad28" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" is_pre_election: true
I20260926 19:39:59.931263 10710 raft_consensus.cc:2454] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 0.
I20260926 19:39:59.931339 10437 raft_consensus.cc:2454] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 0.
I20260926 19:39:59.931777 10503 leader_election.cc:302] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1426574353c44bfab197662b80c87417, 170d8a6a16b642aba38cd6fb1762151e; no voters:
I20260926 19:39:59.932080 10792 raft_consensus.cc:2781] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:39:59.932207 10792 raft_consensus.cc:482] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:39:59.932291 10792 raft_consensus.cc:3037] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:59.934398 10792 raft_consensus.cc:504] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } }
I20260926 19:39:59.934902 10792 leader_election.cc:288] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 election: Requested vote from peers c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579), 1426574353c44bfab197662b80c87417 (127.2.164.67:35139)
I20260926 19:39:59.935283 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892927b1cf504214b967bf7d1469ad28" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc"
I20260926 19:39:59.935340 10710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892927b1cf504214b967bf7d1469ad28" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1426574353c44bfab197662b80c87417"
I20260926 19:39:59.935437 10437 raft_consensus.cc:3037] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:59.935510 10710 raft_consensus.cc:3037] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:39:59.937518 10437 raft_consensus.cc:2454] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 1.
I20260926 19:39:59.937553 10710 raft_consensus.cc:2454] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 1.
I20260926 19:39:59.937841 10503 leader_election.cc:302] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170d8a6a16b642aba38cd6fb1762151e, c2dadcd87d4a4d5ea8cc8797b9ea6bcc; no voters:
I20260926 19:39:59.938086 10792 raft_consensus.cc:2781] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:39:59.938463 10792 raft_consensus.cc:686] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 1 LEADER]: Becoming Leader. State: Replica: 170d8a6a16b642aba38cd6fb1762151e, State: Running, Role: LEADER
I20260926 19:39:59.938858 10792 consensus_queue.cc:237] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } }
I20260926 19:39:59.942910 10276 catalog_manager.cc:5495] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e reported cstate change: term changed from 0 to 1, leader changed from <none> to 170d8a6a16b642aba38cd6fb1762151e (127.2.164.66). New cstate: current_term: 1 leader_uuid: "170d8a6a16b642aba38cd6fb1762151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } health_report { overall_health: HEALTHY } } }
W20260926 19:39:59.985560 10761 tablet.cc:2369] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:39:59.988765 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:39:59.983847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.2.164.66:32951)
0926 19:39:59.984044 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 19:39:59.984050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:39:59.984069 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:39:59.984187 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:39:59.984193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:39:59.984284 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:39:59.984410 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:39:59.984754 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.985599 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.986881 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:39:59.987261 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:39:59.987357 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:39:59.987726 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:39:59.988245 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:39:59.988363 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 19:39:59.988445 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 19:39:59.988513 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:39:59.996424 10437 raft_consensus.cc:1261] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Refusing update from remote peer 170d8a6a16b642aba38cd6fb1762151e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:59.996424 10710 raft_consensus.cc:1261] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Refusing update from remote peer 170d8a6a16b642aba38cd6fb1762151e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:39:59.997349 10796 consensus_queue.cc:1035] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:39:59.997701 10792 consensus_queue.cc:1035] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:40:00.007048 10799 mvcc.cc:205] Tried to move back new op lower bound from 7333689753576042496 to 7333689753352753152. Current Snapshot: MvccSnapshot[applied={T|T < 7333689753576042496}]
I20260926 19:40:00.010982 10800 mvcc.cc:205] Tried to move back new op lower bound from 7333689753576042496 to 7333689753352753152. Current Snapshot: MvccSnapshot[applied={T|T < 7333689753576042496}]
I20260926 19:40:00.021152 10275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:40:00.021453 10275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:40:00.024829 10275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882: Sending DeleteTablet for 3 replicas of tablet 892927b1cf504214b967bf7d1469ad28
I20260926 19:40:00.025835 10417 tablet_service.cc:1514] Processing DeleteTablet for tablet 892927b1cf504214b967bf7d1469ad28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:00 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:48678
I20260926 19:40:00.025990 10551 tablet_service.cc:1514] Processing DeleteTablet for tablet 892927b1cf504214b967bf7d1469ad28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:00 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:57174
I20260926 19:40:00.026873 10690 tablet_service.cc:1514] Processing DeleteTablet for tablet 892927b1cf504214b967bf7d1469ad28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:00 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:43978
I20260926 19:40:00.028867 10275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:40:00.030447 10275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:40:00.039532 10811 tablet_replica.cc:330] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: stopping tablet replica
I20260926 19:40:00.039892 10550 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb86fde6012453f9e4542d3d6b59c7e (DEFAULT_TABLE table=test-table [id=1845df3c48d341b7b86360e0621a8497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:00.040305 10416 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb86fde6012453f9e4542d3d6b59c7e (DEFAULT_TABLE table=test-table [id=1845df3c48d341b7b86360e0621a8497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:00.040503 10550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb86fde6012453f9e4542d3d6b59c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.040624 10690 tablet_service.cc:1467] Processing CreateTablet for tablet 1eb86fde6012453f9e4542d3d6b59c7e (DEFAULT_TABLE table=test-table [id=1845df3c48d341b7b86360e0621a8497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:00.040794 10416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb86fde6012453f9e4542d3d6b59c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.040999 10690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eb86fde6012453f9e4542d3d6b59c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.046851 10786 tablet_bootstrap.cc:492] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Bootstrap starting.
I20260926 19:40:00.047447 10811 raft_consensus.cc:2229] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:40:00.047964 10785 tablet_bootstrap.cc:492] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e: Bootstrap starting.
I20260926 19:40:00.048362 10786 tablet_bootstrap.cc:654] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.050222 10785 tablet_bootstrap.cc:654] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.057197 10811 raft_consensus.cc:2258] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:00.063333 10787 tablet_bootstrap.cc:492] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417: Bootstrap starting.
I20260926 19:40:00.063752 10785 tablet_bootstrap.cc:492] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e: No bootstrap required, opened a new log
I20260926 19:40:00.063937 10785 ts_tablet_manager.cc:1397] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260926 19:40:00.064620 10810 tablet_replica.cc:330] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e: stopping tablet replica
I20260926 19:40:00.064874 10787 tablet_bootstrap.cc:654] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.064777 10785 raft_consensus.cc:348] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.065079 10785 raft_consensus.cc:374] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.065223 10785 raft_consensus.cc:729] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170d8a6a16b642aba38cd6fb1762151e, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.065529 10811 ts_tablet_manager.cc:1905] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:00.065503 10785 consensus_queue.cc:260] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.065630 10809 tablet_replica.cc:330] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: stopping tablet replica
I20260926 19:40:00.066074 10785 ts_tablet_manager.cc:1428] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:40:00.066058 10809 raft_consensus.cc:2229] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:40:00.066521 10809 raft_consensus.cc:2258] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:00.065078 10810 raft_consensus.cc:2229] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:40:00.067313 10786 tablet_bootstrap.cc:492] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: No bootstrap required, opened a new log
I20260926 19:40:00.067466 10786 ts_tablet_manager.cc:1397] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20260926 19:40:00.068245 10786 raft_consensus.cc:348] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.068605 10786 raft_consensus.cc:374] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.069492 10810 raft_consensus.cc:2258] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:00.069833 10786 raft_consensus.cc:729] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dadcd87d4a4d5ea8cc8797b9ea6bcc, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.070215 10809 ts_tablet_manager.cc:1905] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:00.071358 10787 tablet_bootstrap.cc:492] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417: No bootstrap required, opened a new log
I20260926 19:40:00.070216 10786 consensus_queue.cc:260] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.071506 10787 ts_tablet_manager.cc:1397] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260926 19:40:00.071920 10786 ts_tablet_manager.cc:1428] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 19:40:00.072288 10787 raft_consensus.cc:348] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.072559 10787 raft_consensus.cc:374] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.072660 10787 raft_consensus.cc:729] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1426574353c44bfab197662b80c87417, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.072908 10787 consensus_queue.cc:260] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.073583 10787 ts_tablet_manager.cc:1428] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 19:40:00.075409 10810 ts_tablet_manager.cc:1905] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:00.077863 10811 ts_tablet_manager.cc:1918] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:40:00.078022 10811 log.cc:1196] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/892927b1cf504214b967bf7d1469ad28
I20260926 19:40:00.078508 10811 ts_tablet_manager.cc:1939] T 892927b1cf504214b967bf7d1469ad28 P 1426574353c44bfab197662b80c87417: Deleting consensus metadata
I20260926 19:40:00.079303 10809 ts_tablet_manager.cc:1918] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:40:00.079432 10809 log.cc:1196] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/892927b1cf504214b967bf7d1469ad28
I20260926 19:40:00.079815 10809 ts_tablet_manager.cc:1939] T 892927b1cf504214b967bf7d1469ad28 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting consensus metadata
I20260926 19:40:00.080132 10261 catalog_manager.cc:4841] TS 1426574353c44bfab197662b80c87417 (127.2.164.67:35139): tablet 892927b1cf504214b967bf7d1469ad28 (table test-table [id=4112bc9097f94342ae63b5035958c2d7]) successfully deleted
I20260926 19:40:00.081157 10261 catalog_manager.cc:4841] TS c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579): tablet 892927b1cf504214b967bf7d1469ad28 (table test-table [id=4112bc9097f94342ae63b5035958c2d7]) successfully deleted
I20260926 19:40:00.086812 10810 ts_tablet_manager.cc:1918] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:40:00.086968 10810 log.cc:1196] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/892927b1cf504214b967bf7d1469ad28
I20260926 19:40:00.087443 10810 ts_tablet_manager.cc:1939] T 892927b1cf504214b967bf7d1469ad28 P 170d8a6a16b642aba38cd6fb1762151e: Deleting consensus metadata
I20260926 19:40:00.088948 10263 catalog_manager.cc:4841] TS 170d8a6a16b642aba38cd6fb1762151e (127.2.164.66:32951): tablet 892927b1cf504214b967bf7d1469ad28 (table test-table [id=4112bc9097f94342ae63b5035958c2d7]) successfully deleted
I20260926 19:40:00.208734 10792 raft_consensus.cc:482] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:40:00.208935 10792 raft_consensus.cc:504] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.209647 10792 leader_election.cc:288] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579), 1426574353c44bfab197662b80c87417 (127.2.164.67:35139)
I20260926 19:40:00.210191 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb86fde6012453f9e4542d3d6b59c7e" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" is_pre_election: true
I20260926 19:40:00.210222 10710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb86fde6012453f9e4542d3d6b59c7e" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1426574353c44bfab197662b80c87417" is_pre_election: true
I20260926 19:40:00.210449 10437 raft_consensus.cc:2454] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 0.
I20260926 19:40:00.210454 10710 raft_consensus.cc:2454] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 0.
I20260926 19:40:00.210845 10503 leader_election.cc:302] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170d8a6a16b642aba38cd6fb1762151e, c2dadcd87d4a4d5ea8cc8797b9ea6bcc; no voters:
I20260926 19:40:00.211102 10792 raft_consensus.cc:2781] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:40:00.211234 10792 raft_consensus.cc:482] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:40:00.211331 10792 raft_consensus.cc:3037] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:00.213523 10792 raft_consensus.cc:504] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.214098 10792 leader_election.cc:288] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 election: Requested vote from peers c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579), 1426574353c44bfab197662b80c87417 (127.2.164.67:35139)
I20260926 19:40:00.214582 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb86fde6012453f9e4542d3d6b59c7e" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc"
I20260926 19:40:00.214634 10710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eb86fde6012453f9e4542d3d6b59c7e" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1426574353c44bfab197662b80c87417"
I20260926 19:40:00.214756 10437 raft_consensus.cc:3037] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:00.214831 10710 raft_consensus.cc:3037] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:00.216670 10437 raft_consensus.cc:2454] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 1.
I20260926 19:40:00.217073 10503 leader_election.cc:302] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170d8a6a16b642aba38cd6fb1762151e, c2dadcd87d4a4d5ea8cc8797b9ea6bcc; no voters:
I20260926 19:40:00.217387 10792 raft_consensus.cc:2781] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:40:00.217514 10792 raft_consensus.cc:686] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 1 LEADER]: Becoming Leader. State: Replica: 170d8a6a16b642aba38cd6fb1762151e, State: Running, Role: LEADER
I20260926 19:40:00.217754 10792 consensus_queue.cc:237] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.218053 10710 raft_consensus.cc:2454] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 1.
I20260926 19:40:00.222826 10275 catalog_manager.cc:5495] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e reported cstate change: term changed from 0 to 1, leader changed from <none> to 170d8a6a16b642aba38cd6fb1762151e (127.2.164.66). New cstate: current_term: 1 leader_uuid: "170d8a6a16b642aba38cd6fb1762151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } health_report { overall_health: UNKNOWN } } }
W20260926 19:40:00.257812 10622 tablet.cc:2369] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 19:40:00 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451597, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.126@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
W20260926 19:40:00.298928 10483 tablet.cc:2369] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:40:00.299726 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.271192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.2.164.126:37235)
0926 19:40:00.282874 (+ 11682us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:00.282880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:00.283173 (+ 293us) server_negotiation.cc:407] Connection header received
0926 19:40:00.283314 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:00.283319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:00.283399 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:00.283561 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:00.284701 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.285910 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.287108 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.287708 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.287827 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:00.294194 (+ 6367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:00.294199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:00.294202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:00.294229 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:00.296974 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:00.297802 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:00.297806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:00.297808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:00.297924 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:00.298315 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:00.298318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:00.298320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:00.298565 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:00.298663 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:00.299276 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 19:40:00.299438 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11492,"thread_start_us":11295,"threads_started":1}
I20260926 19:40:00.299778 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.271143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:37235 (local address 127.0.0.1:59924)
0926 19:40:00.283062 (+ 11919us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:00.283079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:00.283206 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:00.283575 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:00.283578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:00.284287 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:00.284539 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:00.284548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.286031 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.286035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:00.286960 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:00.286970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.287866 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.287871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:00.287995 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:00.291673 (+ 3678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:00.291694 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:00.293984 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:00.297099 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:00.297111 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:00.297119 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:00.297688 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:00.298026 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:00.298031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:00.298035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:00.298215 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:00.298689 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:00.298698 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:00.298898 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 19:40:00.299069 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 19:40:00.299333 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11780,"thread_start_us":11614,"threads_started":1}
I20260926 19:40:00.304433 10275 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:59924:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 19:40:00.305449 10275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:40:00.311735 10417 tablet_service.cc:1467] Processing CreateTablet for tablet 31141b84f69647c4bff5ed19ec53deba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c901547759454745b87b191fd0317f8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:40:00.312043 10690 tablet_service.cc:1467] Processing CreateTablet for tablet 31141b84f69647c4bff5ed19ec53deba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c901547759454745b87b191fd0317f8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:40:00.312047 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 31141b84f69647c4bff5ed19ec53deba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c901547759454745b87b191fd0317f8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:40:00.312294 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31141b84f69647c4bff5ed19ec53deba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.312527 10690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31141b84f69647c4bff5ed19ec53deba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.312553 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31141b84f69647c4bff5ed19ec53deba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.319453 10785 tablet_bootstrap.cc:492] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: Bootstrap starting.
I20260926 19:40:00.319540 10786 tablet_bootstrap.cc:492] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Bootstrap starting.
I20260926 19:40:00.321054 10786 tablet_bootstrap.cc:654] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.321091 10785 tablet_bootstrap.cc:654] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.322510 10787 tablet_bootstrap.cc:492] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417: Bootstrap starting.
I20260926 19:40:00.324086 10787 tablet_bootstrap.cc:654] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.331363 10785 tablet_bootstrap.cc:492] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: No bootstrap required, opened a new log
I20260926 19:40:00.331380 10787 tablet_bootstrap.cc:492] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417: No bootstrap required, opened a new log
I20260926 19:40:00.331522 10787 ts_tablet_manager.cc:1397] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 19:40:00.332087 10786 tablet_bootstrap.cc:492] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: No bootstrap required, opened a new log
I20260926 19:40:00.332227 10786 ts_tablet_manager.cc:1397] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260926 19:40:00.332216 10787 raft_consensus.cc:348] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.332432 10787 raft_consensus.cc:374] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.332497 10787 raft_consensus.cc:729] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1426574353c44bfab197662b80c87417, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.332736 10787 consensus_queue.cc:260] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.332857 10785 ts_tablet_manager.cc:1397] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260926 19:40:00.332867 10786 raft_consensus.cc:348] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.333087 10786 raft_consensus.cc:374] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.333211 10786 raft_consensus.cc:729] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dadcd87d4a4d5ea8cc8797b9ea6bcc, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.333458 10786 consensus_queue.cc:260] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.333616 10785 raft_consensus.cc:348] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.333885 10785 raft_consensus.cc:374] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.333988 10785 raft_consensus.cc:729] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170d8a6a16b642aba38cd6fb1762151e, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.333915 10793 tablet_replica.cc:439] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } }
I20260926 19:40:00.334259 10793 tablet_replica.cc:442] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:40:00.334259 10785 consensus_queue.cc:260] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.334707 10786 ts_tablet_manager.cc:1428] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:40:00.334931 10785 ts_tablet_manager.cc:1428] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:40:00.335506 10787 ts_tablet_manager.cc:1428] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260926 19:40:00.335381 10791 tablet_replica.cc:439] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } }
I20260926 19:40:00.335743 10791 tablet_replica.cc:442] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:40:00.334785 10792 tablet_replica.cc:439] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } }
I20260926 19:40:00.336997 10792 tablet_replica.cc:442] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:40:00.435740 10792 raft_consensus.cc:482] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:40:00.435945 10792 raft_consensus.cc:504] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.436519 10792 leader_election.cc:288] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579), 1426574353c44bfab197662b80c87417 (127.2.164.67:35139)
I20260926 19:40:00.436982 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31141b84f69647c4bff5ed19ec53deba" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" is_pre_election: true
I20260926 19:40:00.437090 10710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31141b84f69647c4bff5ed19ec53deba" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1426574353c44bfab197662b80c87417" is_pre_election: true
I20260926 19:40:00.437233 10437 raft_consensus.cc:2454] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 0.
I20260926 19:40:00.437384 10710 raft_consensus.cc:2454] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 0.
I20260926 19:40:00.437639 10503 leader_election.cc:302] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170d8a6a16b642aba38cd6fb1762151e, c2dadcd87d4a4d5ea8cc8797b9ea6bcc; no voters:
I20260926 19:40:00.437913 10792 raft_consensus.cc:2781] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:40:00.438012 10792 raft_consensus.cc:482] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:40:00.438110 10792 raft_consensus.cc:3037] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:00.440973 10792 raft_consensus.cc:504] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.441617 10792 leader_election.cc:288] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 election: Requested vote from peers c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579), 1426574353c44bfab197662b80c87417 (127.2.164.67:35139)
I20260926 19:40:00.441970 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31141b84f69647c4bff5ed19ec53deba" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc"
I20260926 19:40:00.442190 10437 raft_consensus.cc:3037] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:00.442104 10710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31141b84f69647c4bff5ed19ec53deba" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1426574353c44bfab197662b80c87417"
I20260926 19:40:00.442333 10710 raft_consensus.cc:3037] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:00.444972 10437 raft_consensus.cc:2454] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 1.
I20260926 19:40:00.444973 10710 raft_consensus.cc:2454] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 1.
I20260926 19:40:00.445374 10503 leader_election.cc:302] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1426574353c44bfab197662b80c87417, 170d8a6a16b642aba38cd6fb1762151e; no voters:
I20260926 19:40:00.445614 10792 raft_consensus.cc:2781] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:40:00.445782 10792 raft_consensus.cc:686] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [term 1 LEADER]: Becoming Leader. State: Replica: 170d8a6a16b642aba38cd6fb1762151e, State: Running, Role: LEADER
I20260926 19:40:00.446110 10792 consensus_queue.cc:237] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.447602 10796 tablet_replica.cc:439] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: TxnStatusTablet state changed. Reason: New leader 170d8a6a16b642aba38cd6fb1762151e. Latest consensus state: current_term: 1 leader_uuid: "170d8a6a16b642aba38cd6fb1762151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } }
I20260926 19:40:00.447849 10796 tablet_replica.cc:442] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:40:00.448417 10827 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 19:40:00.448647 10827 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 19:40:00.449059 10274 catalog_manager.cc:5495] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e reported cstate change: term changed from 0 to 1, leader changed from <none> to 170d8a6a16b642aba38cd6fb1762151e (127.2.164.66). New cstate: current_term: 1 leader_uuid: "170d8a6a16b642aba38cd6fb1762151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } health_report { overall_health: UNKNOWN } } }
I20260926 19:40:00.464409 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.459249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.2.164.66:32951)
0926 19:40:00.459403 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:00.459407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:00.459447 (+ 40us) server_negotiation.cc:407] Connection header received
0926 19:40:00.459694 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:00.459697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:00.459759 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:00.459827 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:40:00.460210 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.460969 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.462829 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.463224 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.463315 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:00.463773 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:40:00.463902 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:40:00.463996 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:00.464125 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 19:40:00.464184 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:40:00.464534 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.459165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32951 (local address 127.0.0.1:57184)
0926 19:40:00.459369 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:00.459390 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:00.459536 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:00.459872 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:00.459878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:00.459891 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:40:00.460095 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:00.460106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.461090 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.461096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:00.462699 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:00.462714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.463371 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.463378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:00.463557 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:00.463624 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:40:00.464013 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:40:00.464031 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 19:40:00.464112 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 19:40:00.464254 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 19:40:00.467947 10710 raft_consensus.cc:1261] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Refusing update from remote peer 170d8a6a16b642aba38cd6fb1762151e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:40:00.467957 10437 raft_consensus.cc:1261] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Refusing update from remote peer 170d8a6a16b642aba38cd6fb1762151e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:40:00.468466 10792 consensus_queue.cc:1035] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:40:00.468755 10796 consensus_queue.cc:1035] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:40:00.470499 10791 tablet_replica.cc:439] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417: TxnStatusTablet state changed. Reason: New leader 170d8a6a16b642aba38cd6fb1762151e. Latest consensus state: current_term: 1 leader_uuid: "170d8a6a16b642aba38cd6fb1762151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } }
I20260926 19:40:00.470731 10791 tablet_replica.cc:442] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:40:00.472848 10796 tablet_replica.cc:439] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170d8a6a16b642aba38cd6fb1762151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } }
I20260926 19:40:00.473109 10796 tablet_replica.cc:442] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:40:00.474864 10801 mvcc.cc:205] Tried to move back new op lower bound from 7333689755511013376 to 7333689755429203968. Current Snapshot: MvccSnapshot[applied={T|T < 7333689755511013376}]
I20260926 19:40:00.475142 10791 tablet_replica.cc:439] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170d8a6a16b642aba38cd6fb1762151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } }
I20260926 19:40:00.475349 10791 tablet_replica.cc:442] T 31141b84f69647c4bff5ed19ec53deba P 1426574353c44bfab197662b80c87417: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:40:00.482511 10793 tablet_replica.cc:439] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: TxnStatusTablet state changed. Reason: New leader 170d8a6a16b642aba38cd6fb1762151e. Latest consensus state: current_term: 1 leader_uuid: "170d8a6a16b642aba38cd6fb1762151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } }
I20260926 19:40:00.482752 10793 tablet_replica.cc:442] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:40:00.484942 10796 tablet_replica.cc:439] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170d8a6a16b642aba38cd6fb1762151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } }
I20260926 19:40:00.485212 10796 tablet_replica.cc:442] T 31141b84f69647c4bff5ed19ec53deba P 170d8a6a16b642aba38cd6fb1762151e: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:40:00.486202 10793 tablet_replica.cc:439] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170d8a6a16b642aba38cd6fb1762151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } } }
I20260926 19:40:00.486438 10793 tablet_replica.cc:442] T 31141b84f69647c4bff5ed19ec53deba P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:40:00.496968 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.491079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:60005 (local address 127.2.164.66:32951)
0926 19:40:00.491215 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:00.491219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:00.491684 (+ 465us) server_negotiation.cc:407] Connection header received
0926 19:40:00.491872 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:00.491879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:00.491938 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:00.492001 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:40:00.492590 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.493342 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.495331 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.495757 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.495843 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:00.496299 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:40:00.496410 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:40:00.496473 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:00.496630 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 19:40:00.496688 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 19:40:00.497253 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.490985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32951 (local address 127.2.164.66:60005)
0926 19:40:00.491567 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:00.491586 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:00.491739 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:00.492045 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:00.492050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:00.492063 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:40:00.492451 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:00.492464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.493466 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.493472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:00.495200 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:00.495214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.495891 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.495897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:00.496077 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:00.496147 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:40:00.496507 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:40:00.496518 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 19:40:00.496597 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 19:40:00.496749 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":255,"threads_started":1}
I20260926 19:40:00.506139 10437 raft_consensus.cc:1261] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Refusing update from remote peer 170d8a6a16b642aba38cd6fb1762151e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:40:00.506312 10709 raft_consensus.cc:1261] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Refusing update from remote peer 170d8a6a16b642aba38cd6fb1762151e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:40:00.506834 10796 consensus_queue.cc:1035] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:40:00.507108 10805 consensus_queue.cc:1035] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:40:00.555130 10275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:40:00.555373 10275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:40:00.558236 10275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882: Sending DeleteTablet for 3 replicas of tablet 1eb86fde6012453f9e4542d3d6b59c7e
I20260926 19:40:00.558895 10417 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eb86fde6012453f9e4542d3d6b59c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:00 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:48678
I20260926 19:40:00.559067 10690 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eb86fde6012453f9e4542d3d6b59c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:00 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:43978
I20260926 19:40:00.559150 10809 tablet_replica.cc:330] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: stopping tablet replica
I20260926 19:40:00.559329 10811 tablet_replica.cc:330] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417: stopping tablet replica
I20260926 19:40:00.559322 10809 raft_consensus.cc:2229] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:40:00.559484 10809 raft_consensus.cc:2258] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:00.559489 10811 raft_consensus.cc:2229] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:40:00.559635 10811 raft_consensus.cc:2258] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:00.560228 10551 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eb86fde6012453f9e4542d3d6b59c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:00 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:57174
I20260926 19:40:00.560484 10810 tablet_replica.cc:330] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e: stopping tablet replica
I20260926 19:40:00.560668 10810 raft_consensus.cc:2229] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:40:00.560714 10811 ts_tablet_manager.cc:1905] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:00.560945 10810 raft_consensus.cc:2258] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:00.561090 10809 ts_tablet_manager.cc:1905] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:00.562230 10810 ts_tablet_manager.cc:1905] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:00.567859 10809 ts_tablet_manager.cc:1918] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:40:00.567859 10811 ts_tablet_manager.cc:1918] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:40:00.568050 10809 log.cc:1196] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/1eb86fde6012453f9e4542d3d6b59c7e
I20260926 19:40:00.568048 10811 log.cc:1196] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/1eb86fde6012453f9e4542d3d6b59c7e
I20260926 19:40:00.568470 10809 ts_tablet_manager.cc:1939] T 1eb86fde6012453f9e4542d3d6b59c7e P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting consensus metadata
I20260926 19:40:00.568503 10811 ts_tablet_manager.cc:1939] T 1eb86fde6012453f9e4542d3d6b59c7e P 1426574353c44bfab197662b80c87417: Deleting consensus metadata
I20260926 19:40:00.569500 10261 catalog_manager.cc:4841] TS c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579): tablet 1eb86fde6012453f9e4542d3d6b59c7e (table test-table [id=1845df3c48d341b7b86360e0621a8497]) successfully deleted
I20260926 19:40:00.569864 10261 catalog_manager.cc:4841] TS 1426574353c44bfab197662b80c87417 (127.2.164.67:35139): tablet 1eb86fde6012453f9e4542d3d6b59c7e (table test-table [id=1845df3c48d341b7b86360e0621a8497]) successfully deleted
I20260926 19:40:00.576524 10810 ts_tablet_manager.cc:1918] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:40:00.576686 10810 log.cc:1196] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/1eb86fde6012453f9e4542d3d6b59c7e
I20260926 19:40:00.577091 10810 ts_tablet_manager.cc:1939] T 1eb86fde6012453f9e4542d3d6b59c7e P 170d8a6a16b642aba38cd6fb1762151e: Deleting consensus metadata
I20260926 19:40:00.578094 10263 catalog_manager.cc:4841] TS 170d8a6a16b642aba38cd6fb1762151e (127.2.164.66:32951): tablet 1eb86fde6012453f9e4542d3d6b59c7e (table test-table [id=1845df3c48d341b7b86360e0621a8497]) successfully deleted
Sep 26 19:40:00 dist-test-slave-mrw4 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 19:40:00 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:00 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 19:40:00.623672 10822 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:00.612671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.2.164.126:37235)
0926 19:40:00.612885 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:00.612892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:00.613302 (+ 410us) server_negotiation.cc:407] Connection header received
0926 19:40:00.613494 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:00.613499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:00.613573 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:00.613655 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:00.615208 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.615913 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.617668 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.618059 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.618144 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:00.623419 (+ 5275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59934: BlockingRecv error: recv got EOF from 127.0.0.1:59934 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 19:40:00 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM
I20260926 19:40:00.640678 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.626672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.2.164.126:37235)
0926 19:40:00.626841 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:00.626847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:00.627220 (+ 373us) server_negotiation.cc:407] Connection header received
0926 19:40:00.627374 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:00.627379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:00.627433 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:00.627505 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:00.628902 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.629734 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.631404 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.631761 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.631847 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:00.635760 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:00.635765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:00.635768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:00.635796 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:00.638099 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:00.638726 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:00.638731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:00.638733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:00.638859 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:00.639206 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:00.639212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:00.639215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:00.639619 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:00.639863 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:00.640142 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 19:40:00.640386 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:40:00.643901 10275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:40:00.644593 10275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:40:00.650740 10417 tablet_service.cc:1467] Processing CreateTablet for tablet 45614520c4194641a672919bcf109db3 (DEFAULT_TABLE table=test-table [id=5bcce76f78484a16967b8ac8a538c72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:00.651171 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45614520c4194641a672919bcf109db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.651094 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 45614520c4194641a672919bcf109db3 (DEFAULT_TABLE table=test-table [id=5bcce76f78484a16967b8ac8a538c72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:00.651460 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45614520c4194641a672919bcf109db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.657298 10786 tablet_bootstrap.cc:492] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Bootstrap starting.
I20260926 19:40:00.657704 10785 tablet_bootstrap.cc:492] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e: Bootstrap starting.
I20260926 19:40:00.658686 10690 tablet_service.cc:1467] Processing CreateTablet for tablet 45614520c4194641a672919bcf109db3 (DEFAULT_TABLE table=test-table [id=5bcce76f78484a16967b8ac8a538c72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:00.658943 10786 tablet_bootstrap.cc:654] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.659070 10690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45614520c4194641a672919bcf109db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.659281 10785 tablet_bootstrap.cc:654] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.665685 10787 tablet_bootstrap.cc:492] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417: Bootstrap starting.
I20260926 19:40:00.667538 10787 tablet_bootstrap.cc:654] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.669884 10787 tablet_bootstrap.cc:492] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417: No bootstrap required, opened a new log
I20260926 19:40:00.670003 10787 ts_tablet_manager.cc:1397] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260926 19:40:00.670658 10785 tablet_bootstrap.cc:492] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e: No bootstrap required, opened a new log
I20260926 19:40:00.670887 10785 ts_tablet_manager.cc:1397] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260926 19:40:00.670711 10787 raft_consensus.cc:348] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.670977 10787 raft_consensus.cc:374] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.671069 10787 raft_consensus.cc:729] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1426574353c44bfab197662b80c87417, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.671309 10787 consensus_queue.cc:260] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.672107 10785 raft_consensus.cc:348] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.672422 10785 raft_consensus.cc:374] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.672569 10785 raft_consensus.cc:729] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170d8a6a16b642aba38cd6fb1762151e, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.672891 10787 ts_tablet_manager.cc:1428] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 19:40:00.672890 10785 consensus_queue.cc:260] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.674067 10785 ts_tablet_manager.cc:1428] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 19:40:00.675721 10786 tablet_bootstrap.cc:492] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: No bootstrap required, opened a new log
I20260926 19:40:00.675894 10786 ts_tablet_manager.cc:1397] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.004s
I20260926 19:40:00.676623 10786 raft_consensus.cc:348] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.676832 10786 raft_consensus.cc:374] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.676919 10786 raft_consensus.cc:729] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dadcd87d4a4d5ea8cc8797b9ea6bcc, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.677116 10786 consensus_queue.cc:260] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.677610 10786 ts_tablet_manager.cc:1428] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 19:40:00.832787 10796 raft_consensus.cc:482] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:40:00.832991 10796 raft_consensus.cc:504] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.833588 10796 leader_election.cc:288] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579), 1426574353c44bfab197662b80c87417 (127.2.164.67:35139)
I20260926 19:40:00.834141 10710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45614520c4194641a672919bcf109db3" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1426574353c44bfab197662b80c87417" is_pre_election: true
I20260926 19:40:00.834160 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45614520c4194641a672919bcf109db3" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" is_pre_election: true
I20260926 19:40:00.834371 10710 raft_consensus.cc:2454] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 0.
I20260926 19:40:00.834406 10437 raft_consensus.cc:2454] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 0.
I20260926 19:40:00.834846 10503 leader_election.cc:302] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170d8a6a16b642aba38cd6fb1762151e, c2dadcd87d4a4d5ea8cc8797b9ea6bcc; no voters:
I20260926 19:40:00.835112 10796 raft_consensus.cc:2781] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:40:00.835212 10796 raft_consensus.cc:482] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:40:00.835307 10796 raft_consensus.cc:3037] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:00.838279 10796 raft_consensus.cc:504] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.838814 10796 leader_election.cc:288] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 election: Requested vote from peers c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579), 1426574353c44bfab197662b80c87417 (127.2.164.67:35139)
I20260926 19:40:00.839222 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45614520c4194641a672919bcf109db3" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc"
I20260926 19:40:00.839284 10710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45614520c4194641a672919bcf109db3" candidate_uuid: "170d8a6a16b642aba38cd6fb1762151e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1426574353c44bfab197662b80c87417"
I20260926 19:40:00.839423 10437 raft_consensus.cc:3037] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:00.839509 10710 raft_consensus.cc:3037] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:00.841542 10710 raft_consensus.cc:2454] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 1.
I20260926 19:40:00.841890 10503 leader_election.cc:302] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1426574353c44bfab197662b80c87417, 170d8a6a16b642aba38cd6fb1762151e; no voters:
I20260926 19:40:00.842121 10796 raft_consensus.cc:2781] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:40:00.842133 10437 raft_consensus.cc:2454] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170d8a6a16b642aba38cd6fb1762151e in term 1.
I20260926 19:40:00.842305 10796 raft_consensus.cc:686] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 1 LEADER]: Becoming Leader. State: Replica: 170d8a6a16b642aba38cd6fb1762151e, State: Running, Role: LEADER
I20260926 19:40:00.842607 10796 consensus_queue.cc:237] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.845212 10274 catalog_manager.cc:5495] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e reported cstate change: term changed from 0 to 1, leader changed from <none> to 170d8a6a16b642aba38cd6fb1762151e (127.2.164.66). New cstate: current_term: 1 leader_uuid: "170d8a6a16b642aba38cd6fb1762151e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } health_report { overall_health: UNKNOWN } } }
I20260926 19:40:00.901921 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.895553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.2.164.66:32951)
0926 19:40:00.895723 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:00.895729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:00.895966 (+ 237us) server_negotiation.cc:407] Connection header received
0926 19:40:00.896055 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:00.896060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:00.896140 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:00.896210 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:40:00.896767 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.897823 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.899647 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:00.900267 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:00.900383 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:00.900931 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:40:00.901057 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:40:00.901126 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:00.901557 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 19:40:00.901641 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:40:00.905627 10710 raft_consensus.cc:1261] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Refusing update from remote peer 170d8a6a16b642aba38cd6fb1762151e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:40:00.905633 10437 raft_consensus.cc:1261] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Refusing update from remote peer 170d8a6a16b642aba38cd6fb1762151e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:40:00.906122 10792 consensus_queue.cc:1035] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:40:00.906355 10796 consensus_queue.cc:1035] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:40:00.916503 10275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:40:00.916690 10275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:40:00.919189 10275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882: Sending DeleteTablet for 3 replicas of tablet 45614520c4194641a672919bcf109db3
I20260926 19:40:00.919884 10417 tablet_service.cc:1514] Processing DeleteTablet for tablet 45614520c4194641a672919bcf109db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:00 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:48678
I20260926 19:40:00.919929 10551 tablet_service.cc:1514] Processing DeleteTablet for tablet 45614520c4194641a672919bcf109db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:00 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:57174
I20260926 19:40:00.920013 10690 tablet_service.cc:1514] Processing DeleteTablet for tablet 45614520c4194641a672919bcf109db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:00 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:43978
I20260926 19:40:00.920156 10809 tablet_replica.cc:330] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: stopping tablet replica
I20260926 19:40:00.920261 10811 tablet_replica.cc:330] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417: stopping tablet replica
I20260926 19:40:00.920285 10809 raft_consensus.cc:2229] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:40:00.920431 10809 raft_consensus.cc:2258] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:00.920416 10811 raft_consensus.cc:2229] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:40:00.920634 10811 raft_consensus.cc:2258] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:00.920661 10810 tablet_replica.cc:330] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e: stopping tablet replica
I20260926 19:40:00.920816 10810 raft_consensus.cc:2229] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:40:00.921103 10810 raft_consensus.cc:2258] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:00.921411 10809 ts_tablet_manager.cc:1905] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:00.921548 10811 ts_tablet_manager.cc:1905] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:00.922021 10810 ts_tablet_manager.cc:1905] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:00.925499 10275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:40:00.926304 10275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:40:00.926302 10811 ts_tablet_manager.cc:1918] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:40:00.926415 10811 log.cc:1196] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/45614520c4194641a672919bcf109db3
I20260926 19:40:00.926959 10811 ts_tablet_manager.cc:1939] T 45614520c4194641a672919bcf109db3 P 1426574353c44bfab197662b80c87417: Deleting consensus metadata
I20260926 19:40:00.927340 10809 ts_tablet_manager.cc:1918] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:40:00.927498 10809 log.cc:1196] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/45614520c4194641a672919bcf109db3
I20260926 19:40:00.927858 10809 ts_tablet_manager.cc:1939] T 45614520c4194641a672919bcf109db3 P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting consensus metadata
I20260926 19:40:00.927951 10261 catalog_manager.cc:4841] TS 1426574353c44bfab197662b80c87417 (127.2.164.67:35139): tablet 45614520c4194641a672919bcf109db3 (table test-table [id=5bcce76f78484a16967b8ac8a538c72e]) successfully deleted
I20260926 19:40:00.928680 10810 ts_tablet_manager.cc:1918] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:40:00.928741 10261 catalog_manager.cc:4841] TS c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579): tablet 45614520c4194641a672919bcf109db3 (table test-table [id=5bcce76f78484a16967b8ac8a538c72e]) successfully deleted
I20260926 19:40:00.928797 10810 log.cc:1196] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/45614520c4194641a672919bcf109db3
I20260926 19:40:00.929121 10810 ts_tablet_manager.cc:1939] T 45614520c4194641a672919bcf109db3 P 170d8a6a16b642aba38cd6fb1762151e: Deleting consensus metadata
I20260926 19:40:00.929955 10263 catalog_manager.cc:4841] TS 170d8a6a16b642aba38cd6fb1762151e (127.2.164.66:32951): tablet 45614520c4194641a672919bcf109db3 (table test-table [id=5bcce76f78484a16967b8ac8a538c72e]) successfully deleted
I20260926 19:40:00.932412 10417 tablet_service.cc:1467] Processing CreateTablet for tablet b1aeccb0faa94e30bfd5ac602b240dbd (DEFAULT_TABLE table=test-table [id=1e30bac27e004b5c83a93d91e13fb166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:00.932569 10551 tablet_service.cc:1467] Processing CreateTablet for tablet b1aeccb0faa94e30bfd5ac602b240dbd (DEFAULT_TABLE table=test-table [id=1e30bac27e004b5c83a93d91e13fb166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:00.932765 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1aeccb0faa94e30bfd5ac602b240dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.932915 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1aeccb0faa94e30bfd5ac602b240dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.932852 10690 tablet_service.cc:1467] Processing CreateTablet for tablet b1aeccb0faa94e30bfd5ac602b240dbd (DEFAULT_TABLE table=test-table [id=1e30bac27e004b5c83a93d91e13fb166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:00.933207 10690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1aeccb0faa94e30bfd5ac602b240dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:00.937127 10786 tablet_bootstrap.cc:492] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Bootstrap starting.
I20260926 19:40:00.938602 10786 tablet_bootstrap.cc:654] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.938678 10785 tablet_bootstrap.cc:492] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e: Bootstrap starting.
I20260926 19:40:00.940165 10785 tablet_bootstrap.cc:654] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.941290 10787 tablet_bootstrap.cc:492] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417: Bootstrap starting.
I20260926 19:40:00.942073 10785 tablet_bootstrap.cc:492] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e: No bootstrap required, opened a new log
I20260926 19:40:00.942181 10785 ts_tablet_manager.cc:1397] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260926 19:40:00.942385 10786 tablet_bootstrap.cc:492] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: No bootstrap required, opened a new log
I20260926 19:40:00.942581 10786 ts_tablet_manager.cc:1397] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260926 19:40:00.942617 10787 tablet_bootstrap.cc:654] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:00.942854 10785 raft_consensus.cc:348] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.943095 10785 raft_consensus.cc:374] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.943171 10785 raft_consensus.cc:729] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170d8a6a16b642aba38cd6fb1762151e, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.943243 10786 raft_consensus.cc:348] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.943462 10786 raft_consensus.cc:374] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.943374 10785 consensus_queue.cc:260] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.943528 10786 raft_consensus.cc:729] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dadcd87d4a4d5ea8cc8797b9ea6bcc, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.943807 10786 consensus_queue.cc:260] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.944262 10786 ts_tablet_manager.cc:1428] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 19:40:00.944391 10787 tablet_bootstrap.cc:492] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417: No bootstrap required, opened a new log
I20260926 19:40:00.944519 10787 ts_tablet_manager.cc:1397] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 19:40:00.944746 10785 ts_tablet_manager.cc:1428] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:40:00.945147 10787 raft_consensus.cc:348] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.945406 10787 raft_consensus.cc:374] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:00.945484 10787 raft_consensus.cc:729] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1426574353c44bfab197662b80c87417, State: Initialized, Role: FOLLOWER
I20260926 19:40:00.945709 10787 consensus_queue.cc:260] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.946273 10787 ts_tablet_manager.cc:1428] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 19:40:00.995187 10791 raft_consensus.cc:482] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:40:00.995383 10791 raft_consensus.cc:504] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:00.996964 10791 leader_election.cc:288] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579), 170d8a6a16b642aba38cd6fb1762151e (127.2.164.66:32951)
I20260926 19:40:01.003647 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.997339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:50835 (local address 127.2.164.66:32951)
0926 19:40:00.997484 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:00.997490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:00.998365 (+ 875us) server_negotiation.cc:407] Connection header received
0926 19:40:00.998511 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:00.998515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:00.998575 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:00.998645 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:40:00.999411 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.000252 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.002093 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.002796 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.002941 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.003014 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:01.003282 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 19:40:01.003405 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 19:40:01.003742 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.997237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32951 (local address 127.2.164.67:50835)
0926 19:40:00.998269 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:00.998282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:00.998392 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:00.998666 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:00.998670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:00.998705 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:40:00.999191 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:00.999203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.000375 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.000381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:01.001955 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:01.001966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.002955 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.002960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:01.003132 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.003174 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 19:40:01.003247 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 19:40:01.003360 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":57,"thread_start_us":793,"threads_started":1}
I20260926 19:40:01.003825 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.997257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:34301 (local address 127.2.164.65:41579)
0926 19:40:00.997893 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:00.997898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:00.998246 (+ 348us) server_negotiation.cc:407] Connection header received
0926 19:40:00.998426 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:00.998431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:00.998485 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:00.998563 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:40:00.999218 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.000237 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.002103 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.002906 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.003079 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.003147 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:01.003404 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 19:40:01.003525 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":356,"threads_started":1}
I20260926 19:40:01.004130 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:00.997202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41579 (local address 127.2.164.67:34301)
0926 19:40:00.998158 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:00.998177 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:00.998307 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:00.998587 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:00.998592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:00.998613 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:40:00.998986 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:00.998998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.000363 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.000369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:01.001968 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:01.001980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.003112 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.003119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:01.003270 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.003293 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:40:01.003367 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 19:40:01.003484 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":88,"thread_start_us":628,"threads_started":1}
I20260926 19:40:01.004441 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1aeccb0faa94e30bfd5ac602b240dbd" candidate_uuid: "1426574353c44bfab197662b80c87417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170d8a6a16b642aba38cd6fb1762151e" is_pre_election: true
I20260926 19:40:01.004727 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1aeccb0faa94e30bfd5ac602b240dbd" candidate_uuid: "1426574353c44bfab197662b80c87417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" is_pre_election: true
I20260926 19:40:01.004778 10571 raft_consensus.cc:2454] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1426574353c44bfab197662b80c87417 in term 0.
I20260926 19:40:01.004917 10437 raft_consensus.cc:2454] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1426574353c44bfab197662b80c87417 in term 0.
I20260926 19:40:01.005246 10644 leader_election.cc:302] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1426574353c44bfab197662b80c87417, 170d8a6a16b642aba38cd6fb1762151e; no voters:
I20260926 19:40:01.005573 10791 raft_consensus.cc:2781] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:40:01.005686 10791 raft_consensus.cc:482] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:40:01.005774 10791 raft_consensus.cc:3037] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:01.007671 10791 raft_consensus.cc:504] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:01.008148 10791 leader_election.cc:288] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [CANDIDATE]: Term 1 election: Requested vote from peers c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579), 170d8a6a16b642aba38cd6fb1762151e (127.2.164.66:32951)
I20260926 19:40:01.008512 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1aeccb0faa94e30bfd5ac602b240dbd" candidate_uuid: "1426574353c44bfab197662b80c87417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc"
I20260926 19:40:01.008589 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1aeccb0faa94e30bfd5ac602b240dbd" candidate_uuid: "1426574353c44bfab197662b80c87417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170d8a6a16b642aba38cd6fb1762151e"
I20260926 19:40:01.008687 10437 raft_consensus.cc:3037] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:01.008754 10571 raft_consensus.cc:3037] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:01.010644 10571 raft_consensus.cc:2454] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1426574353c44bfab197662b80c87417 in term 1.
I20260926 19:40:01.010766 10437 raft_consensus.cc:2454] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1426574353c44bfab197662b80c87417 in term 1.
I20260926 19:40:01.011029 10644 leader_election.cc:302] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1426574353c44bfab197662b80c87417, 170d8a6a16b642aba38cd6fb1762151e; no voters:
I20260926 19:40:01.011271 10791 raft_consensus.cc:2781] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:40:01.011670 10791 raft_consensus.cc:686] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 1 LEADER]: Becoming Leader. State: Replica: 1426574353c44bfab197662b80c87417, State: Running, Role: LEADER
I20260926 19:40:01.012046 10791 consensus_queue.cc:237] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } }
I20260926 19:40:01.015174 10275 catalog_manager.cc:5495] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1426574353c44bfab197662b80c87417 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "1426574353c44bfab197662b80c87417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1426574353c44bfab197662b80c87417" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35139 } health_report { overall_health: HEALTHY } } }
I20260926 19:40:01.046705 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:01.041388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.2.164.67:35139)
0926 19:40:01.041891 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:01.041895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:01.041916 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:40:01.042005 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:01.042034 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:01.042110 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:01.042244 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:40:01.042587 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.043379 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.044676 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.045107 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.045219 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.045608 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:40:01.046200 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:40:01.046304 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:01.046383 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 19:40:01.046441 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":288,"threads_started":1}
I20260926 19:40:01.059923 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:01.055422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:54493 (local address 127.2.164.66:32951)
0926 19:40:01.055571 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:01.055575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:01.055915 (+ 340us) server_negotiation.cc:407] Connection header received
0926 19:40:01.056079 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:01.056082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:01.056146 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:01.056214 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:40:01.056597 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.057274 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.058562 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.059000 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.059091 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.059409 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:40:01.059498 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:40:01.059566 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:01.059646 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 19:40:01.059704 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 19:40:01.059983 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:01.055314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:32951 (local address 127.2.164.67:54493)
0926 19:40:01.055837 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:01.055852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:01.055976 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:01.056230 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:01.056233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:01.056242 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:40:01.056479 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:01.056486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.057367 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.057370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:01.058434 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:01.058446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.059128 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.059132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:01.059240 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.059286 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:40:01.059566 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:40:01.059573 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 19:40:01.059626 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 19:40:01.059720 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":238,"threads_started":1}
I20260926 19:40:01.067479 10437 raft_consensus.cc:1261] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Refusing update from remote peer 1426574353c44bfab197662b80c87417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:40:01.067625 10571 raft_consensus.cc:1261] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Refusing update from remote peer 1426574353c44bfab197662b80c87417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:40:01.068125 10862 consensus_queue.cc:1035] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2dadcd87d4a4d5ea8cc8797b9ea6bcc" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:40:01.068511 10791 consensus_queue.cc:1035] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "170d8a6a16b642aba38cd6fb1762151e" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:40:01.090297 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:01.085429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:40835 (local address 127.2.164.67:35139)
0926 19:40:01.085572 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:01.085579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:01.085951 (+ 372us) server_negotiation.cc:407] Connection header received
0926 19:40:01.086092 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:01.086095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:01.086152 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:01.086218 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:40:01.086594 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.087285 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.088873 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.089416 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.089508 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.089800 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:40:01.089904 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:40:01.089964 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:01.090041 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 19:40:01.090095 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 19:40:01.090353 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:01.085352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:35139 (local address 127.2.164.66:40835)
0926 19:40:01.085853 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:01.085870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:01.085987 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:01.086224 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:01.086227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:01.086234 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:40:01.086429 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:01.086436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.087440 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.087445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:01.088729 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:01.088742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.089538 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.089542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:01.089655 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.089688 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:40:01.089967 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:40:01.089973 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 19:40:01.090021 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 19:40:01.090110 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":263,"threads_started":1}
I20260926 19:40:01.119040 10275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:40:01.119242 10275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:40:01.121901 10275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c75419b28ee4f71963f77741ce55882: Sending DeleteTablet for 3 replicas of tablet b1aeccb0faa94e30bfd5ac602b240dbd
I20260926 19:40:01.122624 10551 tablet_service.cc:1514] Processing DeleteTablet for tablet b1aeccb0faa94e30bfd5ac602b240dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:01 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:57174
I20260926 19:40:01.122666 10690 tablet_service.cc:1514] Processing DeleteTablet for tablet b1aeccb0faa94e30bfd5ac602b240dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:01 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:43978
I20260926 19:40:01.122650 10417 tablet_service.cc:1514] Processing DeleteTablet for tablet b1aeccb0faa94e30bfd5ac602b240dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:01 UTC) from {username='oryx', principal='oryx/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:48678
I20260926 19:40:01.122896 10809 tablet_replica.cc:330] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: stopping tablet replica
I20260926 19:40:01.122903 10811 tablet_replica.cc:330] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417: stopping tablet replica
I20260926 19:40:01.123072 10809 raft_consensus.cc:2229] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:40:01.123099 10811 raft_consensus.cc:2229] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:40:01.123235 10809 raft_consensus.cc:2258] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:01.123571 10811 raft_consensus.cc:2258] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:01.123849 10810 tablet_replica.cc:330] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e: stopping tablet replica
I20260926 19:40:01.124027 10810 raft_consensus.cc:2229] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:40:01.124181 10810 raft_consensus.cc:2258] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:01.124289 10809 ts_tablet_manager.cc:1905] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:01.124482 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 10347
I20260926 19:40:01.126277 10811 ts_tablet_manager.cc:1905] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:01.131026 10809 ts_tablet_manager.cc:1918] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:40:01.131237 10809 log.cc:1196] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/b1aeccb0faa94e30bfd5ac602b240dbd
I20260926 19:40:01.131686 10809 ts_tablet_manager.cc:1939] T b1aeccb0faa94e30bfd5ac602b240dbd P c2dadcd87d4a4d5ea8cc8797b9ea6bcc: Deleting consensus metadata
I20260926 19:40:01.132787 10810 ts_tablet_manager.cc:1905] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:01.132860 10261 catalog_manager.cc:4841] TS c2dadcd87d4a4d5ea8cc8797b9ea6bcc (127.2.164.65:41579): tablet b1aeccb0faa94e30bfd5ac602b240dbd (table test-table [id=1e30bac27e004b5c83a93d91e13fb166]) successfully deleted
I20260926 19:40:01.135080 10811 ts_tablet_manager.cc:1918] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:40:01.135289 10811 log.cc:1196] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/b1aeccb0faa94e30bfd5ac602b240dbd
I20260926 19:40:01.135756 10811 ts_tablet_manager.cc:1939] T b1aeccb0faa94e30bfd5ac602b240dbd P 1426574353c44bfab197662b80c87417: Deleting consensus metadata
I20260926 19:40:01.136972 10261 catalog_manager.cc:4841] TS 1426574353c44bfab197662b80c87417 (127.2.164.67:35139): tablet b1aeccb0faa94e30bfd5ac602b240dbd (table test-table [id=1e30bac27e004b5c83a93d91e13fb166]) successfully deleted
I20260926 19:40:01.139308 10810 ts_tablet_manager.cc:1918] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:40:01.139456 10810 log.cc:1196] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/b1aeccb0faa94e30bfd5ac602b240dbd
I20260926 19:40:01.139897 10810 ts_tablet_manager.cc:1939] T b1aeccb0faa94e30bfd5ac602b240dbd P 170d8a6a16b642aba38cd6fb1762151e: Deleting consensus metadata
I20260926 19:40:01.140834 10263 catalog_manager.cc:4841] TS 170d8a6a16b642aba38cd6fb1762151e (127.2.164.66:32951): tablet b1aeccb0faa94e30bfd5ac602b240dbd (table test-table [id=1e30bac27e004b5c83a93d91e13fb166]) successfully deleted
Sep 26 19:40:01 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.65@KRBTEST.COM
I20260926 19:40:01.152370 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:01.125347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.2.164.65:41579)
0926 19:40:01.125595 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:01.125601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:01.137220 (+ 11619us) server_negotiation.cc:407] Connection header received
0926 19:40:01.137398 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:01.137403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:01.137492 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:01.137592 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:01.139140 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.140204 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.142029 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.142462 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.142559 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.147076 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:01.147090 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:01.147105 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:01.147137 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:01.149830 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:01.150466 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:01.150473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:01.150475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:01.150605 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:01.150942 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:01.150945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:01.150947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:01.151279 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:01.151563 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:01.151860 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 19:40:01.152094 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:40:01.153019 10477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:01.260738 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 10347
I20260926 19:40:01.283967 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 10486
Sep 26 19:40:01 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.66@KRBTEST.COM
I20260926 19:40:01.301288 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:01.284806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.2.164.66:32951)
0926 19:40:01.284995 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:01.285001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:01.285125 (+ 124us) server_negotiation.cc:407] Connection header received
0926 19:40:01.285340 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:01.285346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:01.285425 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:01.285517 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:01.287029 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.288126 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.289968 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.290614 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.290730 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.295689 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:01.295707 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:01.295726 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:01.295765 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:01.298725 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:01.299342 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:01.299349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:01.299352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:01.299472 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:01.299831 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:01.299835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:01.299837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:01.300174 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:01.300446 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:01.300747 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 19:40:01.300952 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:40:01.301872 10616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:01.440519 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 10486
I20260926 19:40:01.463723 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 10625
Sep 26 19:40:01 dist-test-slave-mrw4 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.164.67@KRBTEST.COM
I20260926 19:40:01.479135 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:01.464444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.2.164.67:35139)
0926 19:40:01.464915 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:01.464920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:01.464934 (+ 14us) server_negotiation.cc:407] Connection header received
0926 19:40:01.465012 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:01.465016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:01.465079 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:01.465203 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:01.466503 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.467265 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.468447 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:01.468879 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:01.468983 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:01.473387 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:01.473399 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:01.473414 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:01.473452 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:01.476405 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:01.477144 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:01.477152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:01.477154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:01.477404 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:01.477795 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:01.477798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:01.477800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:01.478151 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:01.478367 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:01.478667 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 19:40:01.478825 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
I20260926 19:40:01.479815 10755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:01.613855 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 10625
I20260926 19:40:01.631281 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 10244
I20260926 19:40:01.632359 10316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:01.743432 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 10244
2026-09-26T19:40:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6093 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:40:01 dist-test-slave-mrw4 krb5kdc[10884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:40:01 dist-test-slave-mrw4 krb5kdc[10884](info): set up 2 sockets
Sep 26 19:40:01 dist-test-slave-mrw4 krb5kdc[10884](info): commencing operation
krb5kdc: starting...
W20260926 19:40:02.733809 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:40:02 dist-test-slave-mrw4 krb5kdc[10884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:40:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:40:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:40:02.914865 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33499
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:33499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:40:03.210765 10897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:03.211048 10897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:03.211133 10897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:03.218185 10897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:40:03.218288 10897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:03.218330 10897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:03.218384 10897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:40:03.218427 10897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:40:03.227797 10897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:33499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:03.228942 10897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:03.230108 10897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:03.240866 10903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:03.240923 10902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:03.240948 10905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:03.241835 10897 server_base.cc:1034] running on GCE node
I20260926 19:40:03.242447 10897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:03.243624 10897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:03.244804 10897 hybrid_clock.cc:648] HybridClock initialized: now 1790451603244764 us; error 33 us; skew 500 ppm
I20260926 19:40:03.245107 10897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T19:40:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1568 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:40:03 dist-test-slave-mrw4 krb5kdc[10911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:40:03 dist-test-slave-mrw4 krb5kdc[10911](info): set up 2 sockets
Sep 26 19:40:03 dist-test-slave-mrw4 krb5kdc[10911](info): commencing operation
krb5kdc: starting...
W20260926 19:40:04.313880 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:40:04 dist-test-slave-mrw4 krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:40:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:40:04.499653 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40687
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:40687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:40:04.797086 10924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:04.797389 10924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:04.797528 10924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:04.804374 10924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:40:04.804471 10924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:04.804551 10924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:04.804628 10924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:40:04.804685 10924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:40:04.813836 10924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:40687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:04.814980 10924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:04.816169 10924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:04.826738 10932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:04.826804 10930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:04.826781 10929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:04.827461 10924 server_base.cc:1034] running on GCE node
I20260926 19:40:04.828110 10924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:04.829370 10924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:04.830580 10924 hybrid_clock.cc:648] HybridClock initialized: now 1790451604830536 us; error 36 us; skew 500 ppm
I20260926 19:40:04.830879 10924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:04 dist-test-slave-mrw4 krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:04.833722 10924 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:40:04.835044 10924 webserver.cc:460] Webserver started at http://127.2.164.126:39403/ using document root <none> and password file <none>
I20260926 19:40:04.835668 10924 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:04.835774 10924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:04.836037 10924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:04.838835 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "86227e18c69e4671944ae6c950ea3026"
format_stamp: "Formatted at 2026-09-26 19:40:04 on dist-test-slave-mrw4"
server_key: "a63b9f7f8fa0a140cb98c45f700d3c6e"
server_key_iv: "83a137335cd86587de76cb29d2523a7a"
server_key_version: "encryptionkey@0"
I20260926 19:40:04.839550 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "86227e18c69e4671944ae6c950ea3026"
format_stamp: "Formatted at 2026-09-26 19:40:04 on dist-test-slave-mrw4"
server_key: "a63b9f7f8fa0a140cb98c45f700d3c6e"
server_key_iv: "83a137335cd86587de76cb29d2523a7a"
server_key_version: "encryptionkey@0"
I20260926 19:40:04.844085 10924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 19:40:04.847198 10939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:04.848341 10924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 19:40:04.848493 10924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "86227e18c69e4671944ae6c950ea3026"
format_stamp: "Formatted at 2026-09-26 19:40:04 on dist-test-slave-mrw4"
server_key: "a63b9f7f8fa0a140cb98c45f700d3c6e"
server_key_iv: "83a137335cd86587de76cb29d2523a7a"
server_key_version: "encryptionkey@0"
I20260926 19:40:04.848668 10924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:04.869951 10924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:04.873775 10924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:04.874008 10924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:04.889958 10924 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:40687
I20260926 19:40:04.889979 10990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:40687 every 8 connection(s)
I20260926 19:40:04.891213 10924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:40:04.894605 10991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:04.895496 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 10924
I20260926 19:40:04.895725 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:40:04.896502 2705 external_mini_cluster.cc:1428] Setting key 8c11b555a58a8b6ae1b2ee755a271644
I20260926 19:40:04.907949 10991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026: Bootstrap starting.
Sep 26 19:40:04 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:40:04.911329 10991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:04.912573 10991 log.cc:824] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026: Log is configured to *not* fsync() on all Append() calls
I20260926 19:40:04.915475 10991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026: No bootstrap required, opened a new log
I20260926 19:40:04.919194 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:04.898387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.2.164.126:40687)
0926 19:40:04.899088 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:04.899102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:04.899149 (+ 47us) server_negotiation.cc:407] Connection header received
0926 19:40:04.900884 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:04.900904 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:04.901304 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:04.901614 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:04.903168 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:04.904359 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:04.906052 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:04.906683 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:04.906809 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:04.911070 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:04.911080 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:04.911096 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:04.911140 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:04.914497 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:04.915120 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:04.915129 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:04.915137 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:04.915347 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:04.915935 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:04.915941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:04.915945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:04.916634 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:04.916923 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:04.917639 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 19:40:04.918212 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":304,"threads_started":1}
I20260926 19:40:04.923444 10991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } }
I20260926 19:40:04.923740 10991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:04.923828 10991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86227e18c69e4671944ae6c950ea3026, State: Initialized, Role: FOLLOWER
I20260926 19:40:04.924320 10991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } }
I20260926 19:40:04.924549 10991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:40:04.924671 10991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:40:04.924825 10991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:04.926815 10991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } }
I20260926 19:40:04.927311 10991 leader_election.cc:302] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86227e18c69e4671944ae6c950ea3026; no voters:
I20260926 19:40:04.927896 10991 leader_election.cc:288] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:40:04.927999 10996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:40:04.928570 10996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 1 LEADER]: Becoming Leader. State: Replica: 86227e18c69e4671944ae6c950ea3026, State: Running, Role: LEADER
I20260926 19:40:04.929116 10996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } }
I20260926 19:40:04.929917 10991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:40:04.931711 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86227e18c69e4671944ae6c950ea3026. Latest consensus state: current_term: 1 leader_uuid: "86227e18c69e4671944ae6c950ea3026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } } }
I20260926 19:40:04.932060 10997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86227e18c69e4671944ae6c950ea3026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } } }
I20260926 19:40:04.932061 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [sys.catalog]: This master's current role is: LEADER
I20260926 19:40:04.932370 10997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [sys.catalog]: This master's current role is: LEADER
I20260926 19:40:04.934091 11006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:40:04.938819 11006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:40:04.946938 11006 catalog_manager.cc:1347] Generated new cluster ID: 5d0547800cbb4c578e4bab98dfbd4f55
I20260926 19:40:04.947023 11006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:40:04.960716 11006 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:40:04.961902 11006 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:40:04.967439 11006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026: Generated new TSK 0
I20260926 19:40:04.968171 11006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:40:05.020005 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40687
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:40:05.323246 11017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:05.323480 11017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:05.323592 11017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:05.330560 11017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:05.330655 11017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:05.330796 11017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:40:05.340225 11017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:05.341392 11017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:05.342605 11017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:05.354463 11023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:05.354504 11025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:05.354529 11022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:05.355123 11017 server_base.cc:1034] running on GCE node
I20260926 19:40:05.355626 11017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:05.356472 11017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:05.357643 11017 hybrid_clock.cc:648] HybridClock initialized: now 1790451605357602 us; error 34 us; skew 500 ppm
I20260926 19:40:05.357908 11017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:05 dist-test-slave-mrw4 krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:05.363725 11017 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:40:05.364938 11017 webserver.cc:460] Webserver started at http://127.2.164.65:39647/ using document root <none> and password file <none>
I20260926 19:40:05.365564 11017 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:05.365665 11017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:05.365924 11017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:05.368546 11017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "e727b035cb9e45cd95026e0691d84215"
format_stamp: "Formatted at 2026-09-26 19:40:05 on dist-test-slave-mrw4"
server_key: "4cf8b33eb20194e4fcc7b71c35ec7013"
server_key_iv: "91168886585e21ae557e7277cef15f30"
server_key_version: "encryptionkey@0"
I20260926 19:40:05.369280 11017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "e727b035cb9e45cd95026e0691d84215"
format_stamp: "Formatted at 2026-09-26 19:40:05 on dist-test-slave-mrw4"
server_key: "4cf8b33eb20194e4fcc7b71c35ec7013"
server_key_iv: "91168886585e21ae557e7277cef15f30"
server_key_version: "encryptionkey@0"
I20260926 19:40:05.373739 11017 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 19:40:05.376890 11032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:05.378098 11017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:40:05.378233 11017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "e727b035cb9e45cd95026e0691d84215"
format_stamp: "Formatted at 2026-09-26 19:40:05 on dist-test-slave-mrw4"
server_key: "4cf8b33eb20194e4fcc7b71c35ec7013"
server_key_iv: "91168886585e21ae557e7277cef15f30"
server_key_version: "encryptionkey@0"
I20260926 19:40:05.378388 11017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:05.402935 11017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:05.406773 11017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:05.407012 11017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:05.407847 11017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:05.409072 11017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:05.409152 11017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:05.409283 11017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:05.409334 11017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:05.437525 11017 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:36941
I20260926 19:40:05.437605 11144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:36941 every 8 connection(s)
I20260926 19:40:05.438661 11017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:40:05.447453 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 11017
I20260926 19:40:05.447633 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:40:05.448365 2705 external_mini_cluster.cc:1428] Setting key 66d29914982bbeced6ed9d361fc65a39
Sep 26 19:40:05 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:40:05.459052 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:05.440896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:57735 (local address 127.2.164.126:40687)
0926 19:40:05.441297 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:05.441301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:05.442339 (+ 1038us) server_negotiation.cc:407] Connection header received
0926 19:40:05.444068 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:05.444072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:05.444135 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:05.444235 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:05.445609 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:05.446336 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:05.448223 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:05.448671 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:05.448762 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:05.452649 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:05.452655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:05.452682 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:05.452767 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:05.456160 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:05.456993 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:05.456998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:05.457002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:05.457126 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:05.457523 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:05.457528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:05.457532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:05.457780 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:05.457898 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:05.458509 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 19:40:05.458708 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":177,"threads_started":1}
I20260926 19:40:05.459981 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:05.441460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:57735)
0926 19:40:05.442184 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:05.442232 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:05.443843 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:05.444407 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:05.444414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:05.444893 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:05.445457 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:05.445475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:05.446473 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:05.446477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:05.448090 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:05.448111 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:05.448870 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:05.448878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:05.449062 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:05.449677 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:05.449703 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:05.452384 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:05.456312 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:05.456321 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:05.456335 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:05.456881 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:05.457238 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:05.457243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:05.457247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:05.457416 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:05.457891 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:05.457897 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:05.458192 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 19:40:05.458403 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 19:40:05.458863 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":295,"threads_started":1}
I20260926 19:40:05.461344 11145 heartbeater.cc:347] Connected to a master server at 127.2.164.126:40687
I20260926 19:40:05.461632 11145 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:05.462378 11145 heartbeater.cc:511] Master 127.2.164.126:40687 requested a full tablet report, sending...
I20260926 19:40:05.464530 10956 ts_manager.cc:194] Registered new tserver with Master: e727b035cb9e45cd95026e0691d84215 (127.2.164.65:36941)
I20260926 19:40:05.466154 10956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:57735
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:40:05.494205 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40687
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:40:05.800701 11151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:05.800941 11151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:05.801028 11151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:05.807782 11151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:05.807869 11151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:05.807979 11151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:40:05.816911 11151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:05.818027 11151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:05.819185 11151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:05.830874 11156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:05.830996 11159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:05.830902 11157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:05.831606 11151 server_base.cc:1034] running on GCE node
I20260926 19:40:05.832109 11151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:05.832921 11151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:05.834105 11151 hybrid_clock.cc:648] HybridClock initialized: now 1790451605834057 us; error 43 us; skew 500 ppm
I20260926 19:40:05.834383 11151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:05 dist-test-slave-mrw4 krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:05.837272 11151 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:40:05.838567 11151 webserver.cc:460] Webserver started at http://127.2.164.66:35601/ using document root <none> and password file <none>
I20260926 19:40:05.839150 11151 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:05.839253 11151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:05.839519 11151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:05.842186 11151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "c843a797fe3342b6b388cb7cff68c458"
format_stamp: "Formatted at 2026-09-26 19:40:05 on dist-test-slave-mrw4"
server_key: "9535c0cbcb9f5b81dabdab8f6df6e3de"
server_key_iv: "cd91e532c76673e029f890b2cd494f21"
server_key_version: "encryptionkey@0"
I20260926 19:40:05.842871 11151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "c843a797fe3342b6b388cb7cff68c458"
format_stamp: "Formatted at 2026-09-26 19:40:05 on dist-test-slave-mrw4"
server_key: "9535c0cbcb9f5b81dabdab8f6df6e3de"
server_key_iv: "cd91e532c76673e029f890b2cd494f21"
server_key_version: "encryptionkey@0"
I20260926 19:40:05.847306 11151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 19:40:05.850318 11166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:05.851467 11151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:40:05.851609 11151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "c843a797fe3342b6b388cb7cff68c458"
format_stamp: "Formatted at 2026-09-26 19:40:05 on dist-test-slave-mrw4"
server_key: "9535c0cbcb9f5b81dabdab8f6df6e3de"
server_key_iv: "cd91e532c76673e029f890b2cd494f21"
server_key_version: "encryptionkey@0"
I20260926 19:40:05.851765 11151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:05.875479 11151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:05.879235 11151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:05.879468 11151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:05.880271 11151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:05.881534 11151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:05.881608 11151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:05.881716 11151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:05.881763 11151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:05.909579 11151 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:34901
I20260926 19:40:05.909615 11278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:34901 every 8 connection(s)
I20260926 19:40:05.910955 11151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:40:05.919955 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 11151
I20260926 19:40:05.920106 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:40:05.920647 2705 external_mini_cluster.cc:1428] Setting key bf1feae1e1b571abf09781a547dcc9f4
Sep 26 19:40:05 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:40:05.930079 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:05.913090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:45027 (local address 127.2.164.126:40687)
0926 19:40:05.913277 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:05.913282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:05.914323 (+ 1041us) server_negotiation.cc:407] Connection header received
0926 19:40:05.915948 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:05.915952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:05.916014 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:05.916094 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:05.917420 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:05.918102 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:05.920329 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:05.920898 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:05.921014 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:05.924668 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:05.924673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:05.924677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:05.924706 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:05.927157 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:05.927891 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:05.927895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:05.927898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:05.928018 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:05.928401 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:05.928405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:05.928408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:05.928706 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:05.928852 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:05.929626 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 19:40:05.929795 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:40:05.931201 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:05.913556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:45027)
0926 19:40:05.914183 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:05.914217 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:05.915731 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:05.916264 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:05.916271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:05.916739 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:05.917302 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:05.917321 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:05.918205 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:05.918208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:05.919896 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:05.919911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:05.921299 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:05.921307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:05.921551 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:05.922130 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:05.922150 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:05.924479 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:05.927276 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:05.927284 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:05.927298 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:05.927794 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:05.928103 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:05.928107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:05.928110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:05.928313 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:05.928849 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:05.928856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:05.929262 (+ 406us) client_negotiation.cc:769] Sending connection context
0926 19:40:05.929539 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 19:40:05.930003 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":238,"threads_started":1}
I20260926 19:40:05.932461 11279 heartbeater.cc:347] Connected to a master server at 127.2.164.126:40687
I20260926 19:40:05.932729 11279 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:05.933383 11279 heartbeater.cc:511] Master 127.2.164.126:40687 requested a full tablet report, sending...
I20260926 19:40:05.934651 10956 ts_manager.cc:194] Registered new tserver with Master: c843a797fe3342b6b388cb7cff68c458 (127.2.164.66:34901)
I20260926 19:40:05.935398 10956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:45027
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:40:05.963966 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40687
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:40:06.268959 11284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:06.269227 11284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:06.269315 11284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:06.276109 11284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:06.276201 11284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:06.276314 11284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:40:06.285506 11284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:06.286618 11284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:06.287832 11284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:06.300014 11292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:06.300119 11289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:06.300093 11290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:06.300674 11284 server_base.cc:1034] running on GCE node
I20260926 19:40:06.301285 11284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:06.302117 11284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:06.303323 11284 hybrid_clock.cc:648] HybridClock initialized: now 1790451606303273 us; error 39 us; skew 500 ppm
I20260926 19:40:06.303601 11284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:06 dist-test-slave-mrw4 krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451606, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:06.306455 11284 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:40:06.307698 11284 webserver.cc:460] Webserver started at http://127.2.164.67:34965/ using document root <none> and password file <none>
I20260926 19:40:06.308312 11284 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:06.308418 11284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:06.308698 11284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:06.311470 11284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "03d83e6ea9b441dfb861421dc73bd2cd"
format_stamp: "Formatted at 2026-09-26 19:40:06 on dist-test-slave-mrw4"
server_key: "7ad31a7d548fab61e81d50fbff0bfdc9"
server_key_iv: "53450fac4bd61a68a05a98b9dd35f1b1"
server_key_version: "encryptionkey@0"
I20260926 19:40:06.312268 11284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "03d83e6ea9b441dfb861421dc73bd2cd"
format_stamp: "Formatted at 2026-09-26 19:40:06 on dist-test-slave-mrw4"
server_key: "7ad31a7d548fab61e81d50fbff0bfdc9"
server_key_iv: "53450fac4bd61a68a05a98b9dd35f1b1"
server_key_version: "encryptionkey@0"
I20260926 19:40:06.317124 11284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 19:40:06.320278 11299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:06.321511 11284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:40:06.321640 11284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "03d83e6ea9b441dfb861421dc73bd2cd"
format_stamp: "Formatted at 2026-09-26 19:40:06 on dist-test-slave-mrw4"
server_key: "7ad31a7d548fab61e81d50fbff0bfdc9"
server_key_iv: "53450fac4bd61a68a05a98b9dd35f1b1"
server_key_version: "encryptionkey@0"
I20260926 19:40:06.321806 11284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:06.346347 11284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:06.350412 11284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:06.350663 11284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:06.351570 11284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:06.352871 11284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:06.352944 11284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:06.353058 11284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:06.353107 11284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:06.380388 11284 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:32997
I20260926 19:40:06.380445 11411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:32997 every 8 connection(s)
I20260926 19:40:06.381645 11284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:40:06.390545 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 11284
I20260926 19:40:06.390704 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:40:06.391197 2705 external_mini_cluster.cc:1428] Setting key 50f930577ea5814bc2377ad1d521d7e3
Sep 26 19:40:06 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451606, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:40:06.399987 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:06.384000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:48233 (local address 127.2.164.126:40687)
0926 19:40:06.384182 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:06.384189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:06.385354 (+ 1165us) server_negotiation.cc:407] Connection header received
0926 19:40:06.387040 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:06.387044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:06.387107 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:06.387189 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:06.388541 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:06.389316 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:06.390995 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:06.391424 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:06.391517 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:06.394895 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:06.394900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:06.394903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:06.394935 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:06.397316 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:06.397937 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:06.397943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:06.397946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:06.398055 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:06.398434 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:06.398439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:06.398443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:06.398701 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:06.398824 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:06.399445 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 19:40:06.399645 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:40:06.400810 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:06.384538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:48233)
0926 19:40:06.385213 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:06.385256 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:06.386790 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:06.387362 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:06.387369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:06.387853 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:06.388365 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:06.388381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:06.389427 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:06.389431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:06.390869 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:06.390889 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:06.391624 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:06.391630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:06.391832 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:06.392430 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:06.392451 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:06.394699 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:06.397434 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:06.397442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:06.397457 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:06.397832 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:06.398152 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:06.398157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:06.398161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:06.398336 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:06.398816 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:06.398824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:06.399113 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 19:40:06.399349 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 19:40:06.399798 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":251,"threads_started":1}
I20260926 19:40:06.401973 11412 heartbeater.cc:347] Connected to a master server at 127.2.164.126:40687
I20260926 19:40:06.402225 11412 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:06.402842 11412 heartbeater.cc:511] Master 127.2.164.126:40687 requested a full tablet report, sending...
I20260926 19:40:06.404064 10956 ts_manager.cc:194] Registered new tserver with Master: 03d83e6ea9b441dfb861421dc73bd2cd (127.2.164.67:32997)
I20260926 19:40:06.404872 10956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:48233
I20260926 19:40:06.406083 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab.
Entry for principal oryx/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab.
WARNING: no policy specified for oryx/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.66@KRBTEST.COM" created.
I20260926 19:40:06.468918 11145 heartbeater.cc:503] Master 127.2.164.126:40687 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab.
Entry for principal oryx/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab.
WARNING: no policy specified for oryx/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab.
Entry for principal oryx/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab.
I20260926 19:40:06.514106 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 11017
Sep 26 19:40:06 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:40:06.529135 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:06.515102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.2.164.65:36941)
0926 19:40:06.515667 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:06.515687 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:06.515720 (+ 33us) server_negotiation.cc:407] Connection header received
0926 19:40:06.515918 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:06.515926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:06.516148 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:06.516310 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:06.517681 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:06.518690 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:06.519932 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:06.520482 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:06.520593 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:06.524290 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:06.524301 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:06.524315 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:06.524356 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:06.526934 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:06.527503 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:06.527510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:06.527512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:06.527626 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:06.527919 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:06.527921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:06.527923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:06.528239 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:06.528464 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:06.528666 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 19:40:06.528862 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":251,"threads_started":1}
I20260926 19:40:06.530395 11140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:06.608918 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 11017
I20260926 19:40:06.621622 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 11151
Sep 26 19:40:06 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:40:06.636404 11424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:06.622437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.2.164.66:34901)
0926 19:40:06.622930 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:06.622937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:06.622960 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:40:06.623015 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:06.623021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:06.623229 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:06.623345 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:06.624421 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:06.625471 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:06.626940 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:06.627472 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:06.627561 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:06.631370 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:06.631384 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:06.631399 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:06.631438 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:06.634212 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:06.634811 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:06.634818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:06.634821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:06.634940 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:06.635210 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:06.635213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:06.635215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:06.635510 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:06.635736 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:06.635929 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 19:40:06.636109 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":247,"threads_started":1}
I20260926 19:40:06.637638 11274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:06.717353 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 11151
I20260926 19:40:06.729665 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 11284
Sep 26 19:40:06 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:40:06.744637 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:06.730494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.2.164.67:32997)
0926 19:40:06.731227 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:06.731235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:06.731263 (+ 28us) server_negotiation.cc:407] Connection header received
0926 19:40:06.731320 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:06.731327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:06.731564 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:06.731698 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:06.732837 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:06.733967 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:06.735130 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:06.735651 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:06.735751 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:06.739761 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:06.739772 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:06.739786 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:06.739819 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:06.742443 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:06.743065 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:06.743072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:06.743075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:06.743195 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:06.743460 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:06.743463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:06.743465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:06.743783 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:06.744000 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:06.744194 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 19:40:06.744372 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":430,"threads_started":1}
I20260926 19:40:06.745888 11407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:06.824649 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 11284
I20260926 19:40:06.837013 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 10924
I20260926 19:40:06.838174 10986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:06.938107 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 10924
I20260926 19:40:06.954319 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40687
--webserver_interface=127.2.164.126
--webserver_port=39403
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:40687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:40:07.254302 11429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:07.254581 11429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:07.254667 11429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:07.261642 11429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:40:07.261744 11429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:07.261797 11429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:07.261852 11429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:40:07.261891 11429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:40:07.271350 11429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:40687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=39403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:07.272586 11429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:07.273765 11429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:07.284741 11435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:07.284771 11437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:07.285266 11434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:07.285565 11429 server_base.cc:1034] running on GCE node
I20260926 19:40:07.286283 11429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:07.287626 11429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:07.288827 11429 hybrid_clock.cc:648] HybridClock initialized: now 1790451607288776 us; error 45 us; skew 500 ppm
I20260926 19:40:07.289131 11429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:07 dist-test-slave-mrw4 krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451607, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:07.291949 11429 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:40:07.293367 11429 webserver.cc:460] Webserver started at http://127.2.164.126:39403/ using document root <none> and password file <none>
I20260926 19:40:07.293999 11429 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:07.294121 11429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:07.299116 11429 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 19:40:07.301739 11444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:07.302973 11429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:40:07.303126 11429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "86227e18c69e4671944ae6c950ea3026"
format_stamp: "Formatted at 2026-09-26 19:40:04 on dist-test-slave-mrw4"
server_key: "a63b9f7f8fa0a140cb98c45f700d3c6e"
server_key_iv: "83a137335cd86587de76cb29d2523a7a"
server_key_version: "encryptionkey@0"
I20260926 19:40:07.304023 11429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:07.326833 11429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:07.330240 11429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:07.330448 11429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:07.346805 11429 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:40687
I20260926 19:40:07.346824 11495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:40687 every 8 connection(s)
I20260926 19:40:07.348071 11429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:40:07.351096 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 11429
I20260926 19:40:07.351797 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:36941
--local_ip_for_outbound_sockets=127.2.164.65
--tserver_master_addrs=127.2.164.126:40687
--webserver_port=39647
--webserver_interface=127.2.164.65
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.164.65
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:40:07.354499 11496 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:40:07.357471 11496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026: Bootstrap starting.
I20260926 19:40:07.374963 11496 log.cc:824] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026: Log is configured to *not* fsync() on all Append() calls
I20260926 19:40:07.383029 11496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:40:07.383478 11496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026: Bootstrap complete.
I20260926 19:40:07.391783 11496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } }
I20260926 19:40:07.392577 11496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86227e18c69e4671944ae6c950ea3026, State: Initialized, Role: FOLLOWER
I20260926 19:40:07.393127 11496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } }
I20260926 19:40:07.393384 11496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:40:07.393482 11496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:40:07.393643 11496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:40:07.397146 11496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } }
I20260926 19:40:07.397650 11496 leader_election.cc:302] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86227e18c69e4671944ae6c950ea3026; no voters:
I20260926 19:40:07.398267 11496 leader_election.cc:288] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 19:40:07.398370 11500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 19:40:07.399428 11500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [term 2 LEADER]: Becoming Leader. State: Replica: 86227e18c69e4671944ae6c950ea3026, State: Running, Role: LEADER
I20260926 19:40:07.399989 11500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } }
I20260926 19:40:07.400297 11496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:40:07.402472 11502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86227e18c69e4671944ae6c950ea3026. Latest consensus state: current_term: 2 leader_uuid: "86227e18c69e4671944ae6c950ea3026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } } }
I20260926 19:40:07.402994 11502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [sys.catalog]: This master's current role is: LEADER
I20260926 19:40:07.402549 11501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "86227e18c69e4671944ae6c950ea3026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86227e18c69e4671944ae6c950ea3026" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40687 } } }
I20260926 19:40:07.403139 11501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026 [sys.catalog]: This master's current role is: LEADER
I20260926 19:40:07.415019 11507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:40:07.420228 11507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:40:07.421512 11507 catalog_manager.cc:1259] Loaded cluster ID: 5d0547800cbb4c578e4bab98dfbd4f55
I20260926 19:40:07.421613 11507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:40:07.425240 11507 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:40:07.426569 11507 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 86227e18c69e4671944ae6c950ea3026: Loaded TSK: 0
I20260926 19:40:07.427997 11507 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 19:40:07.667306 11498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:07.667563 11498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:07.667672 11498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:07.674744 11498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:07.674847 11498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:07.674990 11498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:40:07.684342 11498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.65.keytab
--principal=oryx/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:36941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=39647
--tserver_master_addrs=127.2.164.126:40687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:07.685621 11498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:07.686867 11498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:07.699184 11523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:07.699182 11522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:07.703758 11525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:07.705446 11498 server_base.cc:1034] running on GCE node
I20260926 19:40:07.706274 11498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:07.707472 11498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:07.708698 11498 hybrid_clock.cc:648] HybridClock initialized: now 1790451607708621 us; error 61 us; skew 500 ppm
I20260926 19:40:07.709053 11498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:07 dist-test-slave-mrw4 krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451607, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:07.712857 11498 init.cc:377] Logged in from keytab as oryx/127.2.164.65@KRBTEST.COM (short username oryx)
I20260926 19:40:07.714582 11498 webserver.cc:460] Webserver started at http://127.2.164.65:39647/ using document root <none> and password file <none>
I20260926 19:40:07.715462 11498 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:07.715603 11498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:07.722157 11498 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 19:40:07.725215 11532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:07.726684 11498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:40:07.726835 11498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "e727b035cb9e45cd95026e0691d84215"
format_stamp: "Formatted at 2026-09-26 19:40:05 on dist-test-slave-mrw4"
server_key: "4cf8b33eb20194e4fcc7b71c35ec7013"
server_key_iv: "91168886585e21ae557e7277cef15f30"
server_key_version: "encryptionkey@0"
I20260926 19:40:07.728075 11498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:07.762351 11498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:07.767011 11498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:07.767275 11498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:07.768303 11498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:07.769840 11498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:07.769944 11498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:07.770045 11498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:07.770125 11498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:07.797781 11498 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:36941
I20260926 19:40:07.797857 11644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:36941 every 8 connection(s)
I20260926 19:40:07.799082 11498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:40:07.807571 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 11498
I20260926 19:40:07.808142 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:34901
--local_ip_for_outbound_sockets=127.2.164.66
--tserver_master_addrs=127.2.164.126:40687
--webserver_port=35601
--webserver_interface=127.2.164.66
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.164.66
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:40:07 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:07 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 19:40:08.112047 11649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:08.112290 11649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:08.112407 11649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:08.119277 11649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:08.119372 11649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:08.119526 11649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:40:08.129202 11649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.66.keytab
--principal=oryx/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:34901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=35601
--tserver_master_addrs=127.2.164.126:40687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:08.130419 11649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:08.131582 11649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:08.144738 11655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:08.144774 11656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:08.145462 11658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:08.146579 11649 server_base.cc:1034] running on GCE node
I20260926 19:40:08.147311 11649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:08.148470 11649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:08.149710 11649 hybrid_clock.cc:648] HybridClock initialized: now 1790451608149617 us; error 74 us; skew 500 ppm
I20260926 19:40:08.150081 11649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:08 dist-test-slave-mrw4 krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451608, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:08.153990 11649 init.cc:377] Logged in from keytab as oryx/127.2.164.66@KRBTEST.COM (short username oryx)
I20260926 19:40:08.155411 11649 webserver.cc:460] Webserver started at http://127.2.164.66:35601/ using document root <none> and password file <none>
I20260926 19:40:08.156044 11649 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:08.156157 11649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:08.161456 11649 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:40:08.164178 11665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:08.165608 11649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:40:08.165756 11649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "c843a797fe3342b6b388cb7cff68c458"
format_stamp: "Formatted at 2026-09-26 19:40:05 on dist-test-slave-mrw4"
server_key: "9535c0cbcb9f5b81dabdab8f6df6e3de"
server_key_iv: "cd91e532c76673e029f890b2cd494f21"
server_key_version: "encryptionkey@0"
I20260926 19:40:08.166618 11649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:08.201920 11649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:08.206048 11649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:08.206300 11649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:08.207410 11649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:08.208844 11649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:08.208935 11649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:08.209036 11649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:08.209105 11649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:08.245810 11649 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:34901
I20260926 19:40:08.245864 11777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:34901 every 8 connection(s)
I20260926 19:40:08.247210 11649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:40:08.255645 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 11649
I20260926 19:40:08.256201 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:32997
--local_ip_for_outbound_sockets=127.2.164.67
--tserver_master_addrs=127.2.164.126:40687
--webserver_port=34965
--webserver_interface=127.2.164.67
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.164.67
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:40:08 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:08 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 19:40:08.752980 11782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:08.753468 11782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:08.753638 11782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:08.766497 11782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:08.766661 11782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:08.766883 11782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
W20260926 19:40:08.782984 11648 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.972s user 0.000s sys 0.010s
I20260926 19:40:08.784086 11782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/krb5kdc/oryx_127.2.164.67.keytab
--principal=oryx/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:32997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=34965
--tserver_master_addrs=127.2.164.126:40687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
W20260926 19:40:08.784943 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:07.801828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:36931 (local address 127.2.164.126:40687)
0926 19:40:07.802434 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:07.802446 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:07.802784 (+ 338us) server_negotiation.cc:407] Connection header received
0926 19:40:07.804710 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:07.804722 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:07.805034 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:07.805373 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:07.806756 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:07.808145 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:07.809707 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:07.810352 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:07.810507 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:08.784106 (+973599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:36931: BlockingRecv error: recv got EOF from 127.2.164.65:36931 (error 108)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":221,"threads_started":1}
I20260926 19:40:08.785132 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:07.801893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:36931)
0926 19:40:07.802635 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:07.802679 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:07.804279 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:07.805493 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:07.805500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:07.806030 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:07.806585 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:07.806603 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:07.808288 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:07.808293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:07.809572 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:07.809584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:07.810554 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:07.810561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:07.810798 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:07.811374 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:07.811395 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:08.783735 (+972340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":303,"threads_started":1}
I20260926 19:40:08.786242 11782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260926 19:40:08.786320 11645 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:40687 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.164.126:40687: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
I20260926 19:40:08.788231 11782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 26 19:40:08 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:08 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:08.794852 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:08.786963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:50537)
0926 19:40:08.787130 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:08.787149 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:08.787278 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:08.787690 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:08.787696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:08.788029 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:08.788193 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:08.788202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.789229 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.789232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:08.790114 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:08.790126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.790755 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.790761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:08.790919 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:08.791446 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:08.791463 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:08.794585 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 19:40:08.794981 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:08.787264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:50537 (local address 127.2.164.126:40687)
0926 19:40:08.787438 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:08.787443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:08.787456 (+ 13us) server_negotiation.cc:407] Connection header received
0926 19:40:08.787503 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:08.787507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:08.787562 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:08.787641 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:08.788333 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.789071 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.790251 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.790628 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.790737 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:08.794776 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:50537: BlockingRecv error: recv got EOF from 127.2.164.65:50537 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 19:40:08 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:08 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:08.805862 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:08.795621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:36897)
0926 19:40:08.795901 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:08.795914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:08.796008 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:08.796298 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:08.796302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:08.796568 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:08.796719 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:08.796730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.798063 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.798069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:08.799574 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:08.799587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.800573 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.800579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:08.800798 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:08.801624 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:08.801647 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:08.805567 (+ 3920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209}
W20260926 19:40:08.806208 11645 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 19:40:08.806615 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:08.795688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:36897 (local address 127.2.164.126:40687)
0926 19:40:08.795815 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:08.795819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:08.795980 (+ 161us) server_negotiation.cc:407] Connection header received
0926 19:40:08.796106 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:08.796111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:08.796183 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:08.796256 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:08.796856 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.797900 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.799769 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.800422 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.804818 (+ 4396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:08.806362 (+ 1544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:36897: BlockingRecv error: recv got EOF from 127.2.164.65:36897 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 19:40:08.811390 11788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:08.813081 11789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:08.818421 11791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:08.818749 11782 server_base.cc:1034] running on GCE node
I20260926 19:40:08.819288 11782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:08.820145 11782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:08.821373 11782 hybrid_clock.cc:648] HybridClock initialized: now 1790451608821293 us; error 65 us; skew 500 ppm
I20260926 19:40:08.821789 11782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:08 dist-test-slave-mrw4 krb5kdc[10911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451608, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:08.825619 11782 init.cc:377] Logged in from keytab as oryx/127.2.164.67@KRBTEST.COM (short username oryx)
I20260926 19:40:08.827054 11782 webserver.cc:460] Webserver started at http://127.2.164.67:34965/ using document root <none> and password file <none>
I20260926 19:40:08.827713 11782 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:08.827828 11782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:08.833220 11782 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:40:08.835987 11798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:08.837735 11782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:40:08.837877 11782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "03d83e6ea9b441dfb861421dc73bd2cd"
format_stamp: "Formatted at 2026-09-26 19:40:06 on dist-test-slave-mrw4"
server_key: "7ad31a7d548fab61e81d50fbff0bfdc9"
server_key_iv: "53450fac4bd61a68a05a98b9dd35f1b1"
server_key_version: "encryptionkey@0"
I20260926 19:40:08.838763 11782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:08.867136 11782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:08.871181 11782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:08.871436 11782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:08.872349 11782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:08.873616 11782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:08.873706 11782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:08.873806 11782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:08.873878 11782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:08.908336 11782 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:32997
I20260926 19:40:08.908450 11910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:32997 every 8 connection(s)
I20260926 19:40:08.909749 11782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:40:08.918701 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 11782
Sep 26 19:40:08 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:08 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:08.941411 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:08.919618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.2.164.126:40687)
0926 19:40:08.923811 (+ 4193us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:08.923817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:08.924837 (+ 1020us) server_negotiation.cc:407] Connection header received
0926 19:40:08.924994 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:08.925000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:08.925098 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:08.925246 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:08.927035 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.928333 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.930501 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.931225 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.931326 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:08.935371 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:08.935383 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:08.935406 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:08.935445 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:08.938203 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:08.938818 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:08.938825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:08.938832 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:08.938992 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:08.939368 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:08.939371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:08.939373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:08.939804 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:08.940080 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:08.940798 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 19:40:08.941062 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3017,"thread_start_us":3037,"threads_started":1}
W20260926 19:40:09.377216 11781 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.115s user 0.005s sys 0.012s
W20260926 19:40:09.378789 11780 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:08.249766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:50365 (local address 127.2.164.126:40687)
0926 19:40:08.250222 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:08.250228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:08.251387 (+ 1159us) server_negotiation.cc:407] Connection header received
0926 19:40:08.254209 (+ 2822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:08.254214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:08.254282 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:08.254375 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:08.256598 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.257690 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.260281 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.260867 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.260983 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.378454 (+1117471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:50365: BlockingRecv error: recv got EOF from 127.2.164.66:50365 (error 108)
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":209,"threads_started":1}
I20260926 19:40:09.379330 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:08.250307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:50365)
0926 19:40:08.251189 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:08.251247 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:08.253873 (+ 2626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:08.254668 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:08.254682 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:08.255523 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:08.256403 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:08.256430 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.257895 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.257902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:08.260110 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:08.260135 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.261096 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.261103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:08.261481 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:08.262302 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:08.262335 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:09.378236 (+1115901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":301,"threads_started":1}
W20260926 19:40:09.380409 11778 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:40687 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.164.126:40687: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Sep 26 19:40:09 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:09 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 19:40:09.391885 11780 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:09.381069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:55707 (local address 127.2.164.126:40687)
0926 19:40:09.381232 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:09.381237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:09.381346 (+ 109us) server_negotiation.cc:407] Connection header received
0926 19:40:09.381526 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:09.381530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:09.381620 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:09.381695 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:09.382671 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.383406 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.385402 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.385809 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.385907 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.391720 (+ 5813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:55707: BlockingRecv error: recv got EOF from 127.2.164.66:55707 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 19:40:09.391896 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:09.380984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:55707)
0926 19:40:09.381247 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:09.381271 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:09.381419 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:09.381745 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:09.381750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:09.382269 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:09.382510 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:09.382523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.383523 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.383529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:09.385235 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:09.385252 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.385982 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.385988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:09.386199 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.386861 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:09.386886 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:09.391558 (+ 4672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
Sep 26 19:40:09 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:09 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 19:40:09.402117 11780 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:09.392897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:35599 (local address 127.2.164.126:40687)
0926 19:40:09.393084 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:09.393090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:09.393108 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:40:09.393184 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:09.393190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:09.393282 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:09.393394 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:09.393935 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.395138 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.396401 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.397100 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.397265 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.401712 (+ 4447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:35599: BlockingRecv error: recv got EOF from 127.2.164.66:35599 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:40:09.402271 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:09.392650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:35599)
0926 19:40:09.392799 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:09.392816 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:09.392891 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:09.393410 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:09.393413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:09.393659 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:09.393824 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:09.393831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.395309 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.395314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:09.396282 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:09.396293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.397275 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.397280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:09.397416 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.397934 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:09.397952 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:09.402010 (+ 4058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 19:40:09.402701 11778 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 19:40:09 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:09 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 19:40:09.815987 11780 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:09.806889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:60793 (local address 127.2.164.126:40687)
0926 19:40:09.807088 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:09.807102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:09.807433 (+ 331us) server_negotiation.cc:407] Connection header received
0926 19:40:09.807576 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:09.807581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:09.807692 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:09.807776 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:09.808796 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.809652 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.810876 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.811289 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.811392 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.815834 (+ 4442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:60793: BlockingRecv error: recv got EOF from 127.2.164.65:60793 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:40:09.816035 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:09.806681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:60793)
0926 19:40:09.807331 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:09.807347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:09.807474 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:09.807958 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:09.807963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:09.808455 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:09.808651 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:09.808665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.809795 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.809802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:09.810739 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:09.810751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.811452 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.811458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:09.811608 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.812153 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:09.812171 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:09.815667 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":388,"threads_started":1}
W20260926 19:40:09.930850 11913 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.008s user 0.005s sys 0.007s
W20260926 19:40:09.931882 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:08.912180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:56733 (local address 127.2.164.126:40687)
0926 19:40:08.912392 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:08.912398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:08.913623 (+ 1225us) server_negotiation.cc:407] Connection header received
0926 19:40:08.915382 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:08.915389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:08.915483 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:08.915615 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:08.917008 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.918236 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.920514 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.921293 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.921395 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.931674 (+1010279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:56733: BlockingRecv error: recv got EOF from 127.2.164.67:56733 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:40:09.932497 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:08.912682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:56733)
0926 19:40:08.913453 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:08.913504 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:08.915127 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:08.915791 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:08.915819 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:08.916295 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:08.916839 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:08.916856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.918419 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.918424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:08.920308 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:08.920324 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:08.921739 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:08.921747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:08.922053 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:08.922772 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:08.922794 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:09.931510 (+1008716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":299,"threads_started":1}
W20260926 19:40:09.933471 11911 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:40687 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.164.126:40687: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Sep 26 19:40:09 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:09 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:09.942287 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:09.934127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:35261)
0926 19:40:09.934338 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:09.934358 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:09.934478 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:09.934810 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:09.934815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:09.935211 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:09.935377 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:09.935385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.936510 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.936514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:09.937499 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:09.937511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.938260 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.938266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:09.938427 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.938968 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:09.938985 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:09.942038 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 19:40:09.942366 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:09.934209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:35261 (local address 127.2.164.126:40687)
0926 19:40:09.934406 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:09.934412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:09.934429 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:40:09.934587 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:09.934592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:09.934664 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:09.934748 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:09.935497 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.936380 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.937600 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.938090 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.938191 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.942190 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:35261: BlockingRecv error: recv got EOF from 127.2.164.67:35261 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 26 19:40:09 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:09 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:09.950214 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:09.943025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:40911)
0926 19:40:09.943187 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:09.943204 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:09.943298 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:09.943529 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:09.943532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:09.943783 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:09.943915 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:09.943922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.944857 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.944860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:09.945823 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:09.945833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.946446 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.946450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:09.946579 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.947017 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:09.947031 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:09.950023 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 19:40:09.950486 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:09.943081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:40911 (local address 127.2.164.126:40687)
0926 19:40:09.943224 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:09.943228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:09.943242 (+ 14us) server_negotiation.cc:407] Connection header received
0926 19:40:09.943384 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:09.943387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:09.943432 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:09.943500 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:09.944047 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.944754 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.945933 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:09.946342 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:09.946429 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:09.950212 (+ 3783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:40911: BlockingRecv error: recv got EOF from 127.2.164.67:40911 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 19:40:09.950544 11911 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 19:40:10 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:10 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:10.417996 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:10.403247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:42233)
0926 19:40:10.403734 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:10.403749 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:10.403860 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:10.404131 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:10.404134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:10.404528 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:10.404669 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:10.404677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.405698 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.405702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:10.406705 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:10.406717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.407385 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.407390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:10.407512 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:10.414448 (+ 6936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:10.414466 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:10.417774 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":260,"threads_started":1}
W20260926 19:40:10.418175 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:10.403327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:42233 (local address 127.2.164.126:40687)
0926 19:40:10.403483 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:10.403487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:10.403814 (+ 327us) server_negotiation.cc:407] Connection header received
0926 19:40:10.403944 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:10.403948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:10.404018 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:10.404094 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:10.404772 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.405563 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.406816 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.407201 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.407310 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:10.417964 (+ 10654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:42233: BlockingRecv error: recv got EOF from 127.2.164.66:42233 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 19:40:10 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:10 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:10.824965 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:10.816839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:54973)
0926 19:40:10.817335 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:10.817351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:10.817472 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:10.817778 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:10.817783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:10.818274 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:10.818463 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:10.818473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.819426 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.819431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:10.820345 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:10.820357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.821046 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.821052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:10.821225 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:10.821727 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:10.821744 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:10.824657 (+ 2913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
W20260926 19:40:10.825083 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:10.816882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:54973 (local address 127.2.164.126:40687)
0926 19:40:10.817046 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:10.817053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:10.817429 (+ 376us) server_negotiation.cc:407] Connection header received
0926 19:40:10.817596 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:10.817600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:10.817660 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:10.817733 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:10.818599 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.819304 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.820478 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.820911 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.821011 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:10.824860 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:54973: BlockingRecv error: recv got EOF from 127.2.164.65:54973 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 19:40:10 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:10 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:10.959203 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:10.951047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:60829)
0926 19:40:10.951527 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:10.951542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:10.951645 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:10.951921 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:10.951924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:10.952339 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:10.952502 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:10.952512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.953529 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.953533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:10.954456 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:10.954466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.955114 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.955118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:10.955229 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:10.955796 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:10.955812 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:10.958979 (+ 3167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
W20260926 19:40:10.959359 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:10.951209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:60829 (local address 127.2.164.126:40687)
0926 19:40:10.951358 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:10.951362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:10.951609 (+ 247us) server_negotiation.cc:407] Connection header received
0926 19:40:10.951729 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:10.951733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:10.951817 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:10.951895 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:10.952621 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.953416 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.954574 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:10.954980 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:10.955068 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:10.959151 (+ 4083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:60829: BlockingRecv error: recv got EOF from 127.2.164.67:60829 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 19:40:11 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:11 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:11.427095 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:11.418852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:60179)
0926 19:40:11.419265 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:11.419281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:11.419404 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:11.419700 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:11.419705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:11.420114 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:11.420315 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:11.420328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.421421 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.421425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:11.422355 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:11.422367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.423012 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.423015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:11.423142 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:11.423699 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:11.423717 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:11.426868 (+ 3151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":228,"threads_started":1}
W20260926 19:40:11.427227 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:11.418912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:60179 (local address 127.2.164.126:40687)
0926 19:40:11.419118 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:11.419124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:11.419358 (+ 234us) server_negotiation.cc:407] Connection header received
0926 19:40:11.419490 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:11.419493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:11.419550 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:11.419624 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:11.420443 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.421304 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.422495 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.422891 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.422987 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:11.427048 (+ 4061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:60179: BlockingRecv error: recv got EOF from 127.2.164.66:60179 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 19:40:11 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:11 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:11.834548 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:11.825876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:47433)
0926 19:40:11.826631 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:11.826647 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:11.826771 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:11.827098 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:11.827103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:11.827480 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:11.827654 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:11.827664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.828645 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.828648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:11.829590 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:11.829602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.830313 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.830316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:11.830432 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:11.831144 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:11.831162 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:11.834325 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":491,"threads_started":1}
W20260926 19:40:11.834746 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:11.826046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:47433 (local address 127.2.164.126:40687)
0926 19:40:11.826241 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:11.826247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:11.826722 (+ 475us) server_negotiation.cc:407] Connection header received
0926 19:40:11.826874 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:11.826880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:11.826956 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:11.827057 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:11.827774 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.828502 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.829752 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.830203 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.830301 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:11.834502 (+ 4201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:47433: BlockingRecv error: recv got EOF from 127.2.164.65:47433 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 19:40:11 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:11 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:11.968389 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:11.959985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:41871)
0926 19:40:11.960416 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:11.960431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:11.960568 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:11.960871 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:11.960876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:11.961297 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:11.961489 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:11.961501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.962509 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.962513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:11.963409 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:11.963421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.964117 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.964123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:11.964274 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:11.964837 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:11.964855 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:11.968159 (+ 3304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":230,"threads_started":1}
W20260926 19:40:11.968520 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:11.960039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:41871 (local address 127.2.164.126:40687)
0926 19:40:11.960203 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:11.960210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:11.960510 (+ 300us) server_negotiation.cc:407] Connection header received
0926 19:40:11.960678 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:11.960683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:11.960748 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:11.960848 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:11.961625 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.962367 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.963540 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:11.963978 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:11.964081 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:11.968337 (+ 4256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:41871: BlockingRecv error: recv got EOF from 127.2.164.67:41871 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 19:40:12 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:12 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:12.436004 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:12.427908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:53583)
0926 19:40:12.428348 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:12.428363 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:12.428469 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:12.428729 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:12.428732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:12.429057 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:12.429232 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:12.429240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.430162 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.430165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:12.431039 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:12.431048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.431642 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.431646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:12.431757 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:12.432282 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:12.432298 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:12.435677 (+ 3379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":213,"threads_started":1}
W20260926 19:40:12.436108 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:12.427975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:53583 (local address 127.2.164.126:40687)
0926 19:40:12.428174 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:12.428180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:12.428429 (+ 249us) server_negotiation.cc:407] Connection header received
0926 19:40:12.428550 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:12.428553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:12.428627 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:12.428700 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:12.429328 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.430058 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.431134 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.431545 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.431647 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:12.435889 (+ 4242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:53583: BlockingRecv error: recv got EOF from 127.2.164.66:53583 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 19:40:12 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:12 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:12.843839 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:12.835417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:51233)
0926 19:40:12.835906 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:12.835922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:12.836031 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:12.836292 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:12.836294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:12.836636 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:12.836816 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:12.836825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.837844 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.837848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:12.838935 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:12.838947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.839616 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.839622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:12.839785 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:12.840323 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:12.840340 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:12.843492 (+ 3152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":255,"threads_started":1}
W20260926 19:40:12.843896 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:12.835518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:51233 (local address 127.2.164.126:40687)
0926 19:40:12.835675 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:12.835679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:12.836015 (+ 336us) server_negotiation.cc:407] Connection header received
0926 19:40:12.836110 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:12.836114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:12.836176 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:12.836254 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:12.836945 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.837707 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.839060 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.839490 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.839592 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:12.843692 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:51233: BlockingRecv error: recv got EOF from 127.2.164.65:51233 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 19:40:12 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:12 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:12.977154 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:12.969279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:58507)
0926 19:40:12.969744 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:12.969763 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:12.969889 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:12.970131 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:12.970134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:12.970502 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:12.970685 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:12.970694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.971713 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.971716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:12.972593 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:12.972602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.973265 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.973269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:12.973397 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:12.973906 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:12.973922 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:12.976897 (+ 2975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":220,"threads_started":1}
W20260926 19:40:12.977384 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:12.969308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:58507 (local address 127.2.164.126:40687)
0926 19:40:12.969517 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:12.969524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:12.969821 (+ 297us) server_negotiation.cc:407] Connection header received
0926 19:40:12.969969 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:12.969973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:12.970032 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:12.970109 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:12.970850 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.971577 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.972719 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:12.973134 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:12.973256 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:12.977101 (+ 3845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:58507: BlockingRecv error: recv got EOF from 127.2.164.67:58507 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 19:40:13 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:13 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:13.444808 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:13.436967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:59217)
0926 19:40:13.437420 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:13.437435 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:13.437539 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:13.437783 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:13.437786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:13.438122 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:13.438274 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:13.438281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.439140 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.439143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:13.440097 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:13.440107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.440663 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.440666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:13.440788 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:13.441414 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:13.441431 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:13.444580 (+ 3149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":248,"threads_started":1}
W20260926 19:40:13.444978 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:13.437037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:59217 (local address 127.2.164.126:40687)
0926 19:40:13.437280 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:13.437286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:13.437502 (+ 216us) server_negotiation.cc:407] Connection header received
0926 19:40:13.437622 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:13.437625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:13.437683 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:13.437753 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:13.438372 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.439048 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.440195 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.440566 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.440646 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:13.444749 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:59217: BlockingRecv error: recv got EOF from 127.2.164.66:59217 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 19:40:13 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:13 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:13.852882 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:13.844663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:55015)
0926 19:40:13.845158 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:13.845204 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:13.845343 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:13.845641 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:13.845646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:13.846019 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:13.846175 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:13.846184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.847205 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.847211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:13.848128 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:13.848140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.848809 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.848815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:13.848995 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:13.849561 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:13.849578 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:13.852544 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":236,"threads_started":1}
W20260926 19:40:13.852962 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:13.844736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:55015 (local address 127.2.164.126:40687)
0926 19:40:13.844919 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:13.844926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:13.845282 (+ 356us) server_negotiation.cc:407] Connection header received
0926 19:40:13.845456 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:13.845462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:13.845531 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:13.845608 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:13.846279 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.847067 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.848265 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.848674 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.848797 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:13.852747 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:55015: BlockingRecv error: recv got EOF from 127.2.164.65:55015 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 19:40:13 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:13 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:13.986515 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:13.978085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:42905)
0926 19:40:13.978538 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:13.978553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:13.978669 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:13.978979 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:13.978983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:13.979353 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:13.979515 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:13.979524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.980468 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.980473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:13.981469 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:13.981481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.982127 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.982133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:13.982309 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:13.982888 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:13.982906 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:13.986193 (+ 3287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":243,"threads_started":1}
W20260926 19:40:13.986629 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:13.978133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:42905 (local address 127.2.164.126:40687)
0926 19:40:13.978297 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:13.978304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:13.978630 (+ 326us) server_negotiation.cc:407] Connection header received
0926 19:40:13.978778 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:13.978782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:13.978853 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:13.978933 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:13.979653 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.980339 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.981601 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:13.982000 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:13.982104 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:13.986411 (+ 4307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:42905: BlockingRecv error: recv got EOF from 127.2.164.67:42905 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 19:40:14 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:14 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:14.454222 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:14.445644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:34955)
0926 19:40:14.446178 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:14.446195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:14.446322 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:14.446617 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:14.446622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:14.447013 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:14.447171 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:14.447180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.448108 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.448114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:14.449119 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:14.449131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.449840 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.449844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:14.449976 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:14.450488 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:14.450504 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:14.453945 (+ 3441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":293,"threads_started":1}
W20260926 19:40:14.454355 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:14.445901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:34955 (local address 127.2.164.126:40687)
0926 19:40:14.446135 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:14.446142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:14.446270 (+ 128us) server_negotiation.cc:407] Connection header received
0926 19:40:14.446426 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:14.446431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:14.446490 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:14.446564 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:14.447316 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.447991 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.449275 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.449716 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.449814 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:14.454136 (+ 4322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:34955: BlockingRecv error: recv got EOF from 127.2.164.66:34955 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 26 19:40:14 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:14 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:14.861687 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:14.853689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:35931)
0926 19:40:14.854076 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:14.854092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:14.854198 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:14.854447 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:14.854450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:14.854809 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:14.854969 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:14.854977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.855825 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.855828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:14.856813 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:14.856827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.857523 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.857530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:14.857701 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:14.858343 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:14.858365 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:14.861458 (+ 3093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":209,"threads_started":1}
W20260926 19:40:14.861793 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:14.853755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:35931 (local address 127.2.164.126:40687)
0926 19:40:14.853971 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:14.853977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:14.854158 (+ 181us) server_negotiation.cc:407] Connection header received
0926 19:40:14.854281 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:14.854284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:14.854348 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:14.854422 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:14.855068 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.855729 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.856931 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.857390 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.857480 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:14.861611 (+ 4131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:35931: BlockingRecv error: recv got EOF from 127.2.164.65:35931 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 19:40:14 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:14 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:14.995203 11932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:14.987433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:45633)
0926 19:40:14.987875 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:14.987892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:14.988001 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:14.988249 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:14.988252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:14.988639 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:14.988785 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:14.988793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.989731 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.989735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:14.990658 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:14.990669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.991322 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.991326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:14.991451 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:14.991990 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:14.992007 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:14.995002 (+ 2995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":217,"threads_started":1}
W20260926 19:40:14.995383 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:14.987474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:45633 (local address 127.2.164.126:40687)
0926 19:40:14.987656 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:14.987664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:14.987960 (+ 296us) server_negotiation.cc:407] Connection header received
0926 19:40:14.988091 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:14.988095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:14.988154 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:14.988220 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:14.988890 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.989634 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.990774 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:14.991218 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:14.991304 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:14.995163 (+ 3859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:45633: BlockingRecv error: recv got EOF from 127.2.164.67:45633 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 19:40:15 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:15 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:15.468569 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:15.455200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:39953)
0926 19:40:15.455652 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:15.455667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:15.455779 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:15.456013 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:15.456016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:15.456385 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:15.456546 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:15.456554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:15.457595 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:15.457599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:15.458511 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:15.458522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:15.459204 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:15.459211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:15.459367 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:15.465100 (+ 5733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:15.465132 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:15.468277 (+ 3145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":222,"threads_started":1}
W20260926 19:40:15.468701 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:15.455263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:39953 (local address 127.2.164.126:40687)
0926 19:40:15.455476 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:15.455483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:15.455725 (+ 242us) server_negotiation.cc:407] Connection header received
0926 19:40:15.455866 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:15.455870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:15.455928 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:15.455992 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:15.456651 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:15.457466 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:15.458609 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:15.459050 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:15.459140 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:15.468486 (+ 9346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:39953: BlockingRecv error: recv got EOF from 127.2.164.66:39953 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 19:40:15 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:15 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:15.870818 11934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:15.862474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:44043)
0926 19:40:15.862987 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:15.863004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:15.863112 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:15.863362 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:15.863365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:15.863738 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:15.863919 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:15.863928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:15.864890 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:15.864894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:15.865974 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:15.865991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:15.866872 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:15.866877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:15.866990 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:15.867503 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:15.867519 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:15.870576 (+ 3057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":226,"threads_started":1}
W20260926 19:40:15.871065 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:15.862519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:44043 (local address 127.2.164.126:40687)
0926 19:40:15.862733 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:15.862740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:15.863074 (+ 334us) server_negotiation.cc:407] Connection header received
0926 19:40:15.863195 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:15.863198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:15.863256 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:15.863331 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:15.864046 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:15.864781 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:15.866104 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:15.866728 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:15.866815 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:15.870835 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:44043: BlockingRecv error: recv got EOF from 127.2.164.65:44043 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 19:40:15 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:15 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:16.004315 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:15.995994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:45285)
0926 19:40:15.996478 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:15.996493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:15.996662 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:15.996981 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:15.996985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:15.997414 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:15.997586 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:15.997595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:15.998549 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:15.998553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:15.999441 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:15.999451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:16.000155 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:16.000161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:16.000328 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:16.000845 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:16.000862 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:16.004032 (+ 3170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":274,"threads_started":1}
W20260926 19:40:16.004432 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:15.996074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:45285 (local address 127.2.164.126:40687)
0926 19:40:15.996306 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:15.996312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:15.996582 (+ 270us) server_negotiation.cc:407] Connection header received
0926 19:40:15.996781 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:15.996785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:15.996846 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:15.996931 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:15.997726 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:15.998420 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:15.999565 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:16.000018 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:16.000120 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:16.004210 (+ 4090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:45285: BlockingRecv error: recv got EOF from 127.2.164.67:45285 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 19:40:16 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:16 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:16.477665 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:16.469509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:47029)
0926 19:40:16.470201 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:16.470217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:16.470324 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:16.470604 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:16.470608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:16.471021 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:16.471183 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:16.471192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:16.472134 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:16.472137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:16.473053 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:16.473064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:16.473678 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:16.473682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:16.473788 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:16.474294 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:16.474311 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:16.477445 (+ 3134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":357,"threads_started":1}
W20260926 19:40:16.477797 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:16.469584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:47029 (local address 127.2.164.126:40687)
0926 19:40:16.469828 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:16.469835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:16.470272 (+ 437us) server_negotiation.cc:407] Connection header received
0926 19:40:16.470417 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:16.470421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:16.470482 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:16.470554 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:16.471314 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:16.472031 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:16.473152 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:16.473577 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:16.473659 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:16.477609 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:47029: BlockingRecv error: recv got EOF from 127.2.164.66:47029 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 19:40:16 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:16 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:16.880129 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:16.871763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:49533)
0926 19:40:16.872243 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:16.872259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:16.872370 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:16.872735 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:16.872739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:16.873149 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:16.873387 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:16.873397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:16.874422 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:16.874426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:16.875379 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:16.875392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:16.876049 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:16.876054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:16.876183 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:16.876750 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:16.876770 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:16.879901 (+ 3131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":237,"threads_started":1}
W20260926 19:40:16.880286 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:16.871870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:49533 (local address 127.2.164.126:40687)
0926 19:40:16.872026 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:16.872030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:16.872330 (+ 300us) server_negotiation.cc:407] Connection header received
0926 19:40:16.872501 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:16.872505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:16.872565 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:16.872642 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:16.873535 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:16.874268 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:16.875497 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:16.875907 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:16.876001 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:16.880094 (+ 4093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:49533: BlockingRecv error: recv got EOF from 127.2.164.65:49533 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 19:40:17 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:17 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:17.014153 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:17.005297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:43213)
0926 19:40:17.005837 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:17.005853 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:17.005976 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:17.006344 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:17.006348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:17.006828 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:17.007004 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:17.007013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.008078 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.008083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:17.009060 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:17.009073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.009758 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.009763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:17.009876 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:17.010396 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:17.010413 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:17.013839 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":246,"threads_started":1}
W20260926 19:40:17.014252 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:17.005396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:43213 (local address 127.2.164.126:40687)
0926 19:40:17.005545 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:17.005550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:17.005917 (+ 367us) server_negotiation.cc:407] Connection header received
0926 19:40:17.006108 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:17.006112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:17.006173 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:17.006261 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:17.007127 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.007926 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.009204 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.009625 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.009713 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:17.014028 (+ 4315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:43213: BlockingRecv error: recv got EOF from 127.2.164.67:43213 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 19:40:17 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:17 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:17.486824 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:17.478490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:55287)
0926 19:40:17.478932 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:17.478948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:17.479060 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:17.479312 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:17.479315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:17.479722 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:17.479879 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:17.479886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.481247 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.481252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:17.482227 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:17.482240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.482861 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.482865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:17.482976 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:17.483497 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:17.483513 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:17.486510 (+ 2997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":225,"threads_started":1}
W20260926 19:40:17.486923 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:17.478567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:55287 (local address 127.2.164.126:40687)
0926 19:40:17.478761 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:17.478767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:17.479016 (+ 249us) server_negotiation.cc:407] Connection header received
0926 19:40:17.479157 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:17.479161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:17.479219 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:17.479286 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:17.480013 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.481087 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.482351 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.482733 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.482850 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:17.486694 (+ 3844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:55287: BlockingRecv error: recv got EOF from 127.2.164.66:55287 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 19:40:17 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:17 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:17.889093 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:17.881027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:48429)
0926 19:40:17.881468 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:17.881485 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:17.881591 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:17.881841 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:17.881845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:17.882225 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:17.882378 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:17.882386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.883302 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.883305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:17.884330 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:17.884351 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.885112 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.885117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:17.885303 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:17.885862 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:17.885880 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:17.888836 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":243,"threads_started":1}
W20260926 19:40:17.889256 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:17.881074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:48429 (local address 127.2.164.126:40687)
0926 19:40:17.881341 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:17.881347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:17.881553 (+ 206us) server_negotiation.cc:407] Connection header received
0926 19:40:17.881680 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:17.881683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:17.881741 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:17.881816 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:17.882480 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.883207 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.884452 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:17.884974 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:17.885076 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:17.889023 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:48429: BlockingRecv error: recv got EOF from 127.2.164.65:48429 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 26 19:40:18 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:18 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:18.023232 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:18.015128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:48931)
0926 19:40:18.015619 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:18.015635 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:18.015745 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:18.016002 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:18.016005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:18.016381 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:18.016549 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:18.016558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.017479 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.017483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:18.018425 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:18.018436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.019120 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.019124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:18.019231 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:18.019760 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:18.019775 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:18.022915 (+ 3140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":254,"threads_started":1}
W20260926 19:40:18.023308 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:18.015218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:48931 (local address 127.2.164.126:40687)
0926 19:40:18.015363 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:18.015367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:18.015695 (+ 328us) server_negotiation.cc:407] Connection header received
0926 19:40:18.015837 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:18.015841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:18.015897 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:18.015971 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:18.016650 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.017375 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.018539 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.018991 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.019073 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:18.023108 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:48931: BlockingRecv error: recv got EOF from 127.2.164.67:48931 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 19:40:18 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:18 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:18.495709 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:18.487697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:46601)
0926 19:40:18.488132 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:18.488148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:18.488258 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:18.488502 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:18.488504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:18.488899 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:18.489036 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:18.489044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.489935 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.489939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:18.490853 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:18.490864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.491541 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.491545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:18.491655 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:18.492175 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:18.492191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:18.495414 (+ 3223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":232,"threads_started":1}
W20260926 19:40:18.495862 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:18.487740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:46601 (local address 127.2.164.126:40687)
0926 19:40:18.487963 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:18.487969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:18.488217 (+ 248us) server_negotiation.cc:407] Connection header received
0926 19:40:18.488342 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:18.488346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:18.488402 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:18.488472 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:18.489149 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.489833 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.490994 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.491427 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.491518 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:18.495629 (+ 4111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:46601: BlockingRecv error: recv got EOF from 127.2.164.66:46601 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 19:40:18 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:18 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:18.898293 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:18.889908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:57953)
0926 19:40:18.890403 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:18.890419 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:18.890552 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:18.890900 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:18.890904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:18.891316 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:18.891473 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:18.891482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.892437 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.892443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:18.893485 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:18.893497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.894174 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.894180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:18.894331 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:18.894896 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:18.894912 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:18.897959 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":280,"threads_started":1}
W20260926 19:40:18.898396 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:18.890005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:57953 (local address 127.2.164.126:40687)
0926 19:40:18.890190 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:18.890197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:18.890504 (+ 307us) server_negotiation.cc:407] Connection header received
0926 19:40:18.890680 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:18.890685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:18.890756 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:18.890840 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:18.891618 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.892291 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.893618 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:18.894033 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:18.894136 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:18.898154 (+ 4018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:57953: BlockingRecv error: recv got EOF from 127.2.164.65:57953 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 19:40:19 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:19 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:19.032375 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:19.024272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:52877)
0926 19:40:19.024729 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:19.024753 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:19.024872 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:19.025191 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:19.025196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:19.025583 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:19.025718 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:19.025726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.026712 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.026716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:19.027630 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:19.027640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.028308 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.028314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:19.028472 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:19.029019 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:19.029036 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:19.032065 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":229,"threads_started":1}
W20260926 19:40:19.032470 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:19.024341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:52877 (local address 127.2.164.126:40687)
0926 19:40:19.024536 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:19.024542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:19.024850 (+ 308us) server_negotiation.cc:407] Connection header received
0926 19:40:19.024979 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:19.024985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:19.025051 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:19.025126 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:19.025872 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.026587 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.027754 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.028170 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.028276 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:19.032240 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:52877: BlockingRecv error: recv got EOF from 127.2.164.67:52877 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 19:40:19 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:19 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:19.505010 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:19.496558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:45593)
0926 19:40:19.497057 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:19.497071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:19.497222 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:19.497542 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:19.497546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:19.497938 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:19.498089 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:19.498098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.499058 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.499062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:19.500077 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:19.500089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.500813 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.500818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:19.501034 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:19.501629 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:19.501648 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:19.504671 (+ 3023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":271,"threads_started":1}
W20260926 19:40:19.505142 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:19.496668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:45593 (local address 127.2.164.126:40687)
0926 19:40:19.496883 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:19.496890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:19.497185 (+ 295us) server_negotiation.cc:407] Connection header received
0926 19:40:19.497349 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:19.497355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:19.497421 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:19.497497 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:19.498238 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.498937 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.500198 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.500659 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.500763 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:19.504930 (+ 4167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:45593: BlockingRecv error: recv got EOF from 127.2.164.66:45593 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 19:40:19 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:19 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:19.907282 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:19.899191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:34665)
0926 19:40:19.899629 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:19.899645 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:19.899767 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:19.900054 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:19.900058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:19.900473 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:19.900646 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:19.900656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.901717 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.901721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:19.902710 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:19.902725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.903389 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.903394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:19.903542 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:19.904035 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:19.904050 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:19.907071 (+ 3021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":220,"threads_started":1}
W20260926 19:40:19.907490 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:19.899396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:34665 (local address 127.2.164.126:40687)
0926 19:40:19.899585 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:19.899591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:19.899712 (+ 121us) server_negotiation.cc:407] Connection header received
0926 19:40:19.899869 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:19.899874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:19.899940 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:19.900014 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:19.900811 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.901604 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.902842 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:19.903251 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:19.903350 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:19.907247 (+ 3897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:34665: BlockingRecv error: recv got EOF from 127.2.164.65:34665 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 19:40:20 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:20 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:20.041491 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:20.033327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:55325)
0926 19:40:20.033831 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:20.033846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:20.033972 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:20.034286 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:20.034291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:20.034675 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:20.034844 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:20.034852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.035826 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.035832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:20.036777 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:20.036788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.037495 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.037500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:20.037651 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:20.038167 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:20.038183 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:20.041243 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":258,"threads_started":1}
W20260926 19:40:20.041558 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:20.033400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:55325 (local address 127.2.164.126:40687)
0926 19:40:20.033576 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:20.033582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:20.033922 (+ 340us) server_negotiation.cc:407] Connection header received
0926 19:40:20.034079 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:20.034085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:20.034160 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:20.034240 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:20.034977 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.035684 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.036920 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.037361 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.037461 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:20.041390 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:55325: BlockingRecv error: recv got EOF from 127.2.164.67:55325 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 19:40:20 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:20 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:20.519654 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:20.506079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:46909)
0926 19:40:20.506552 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:20.506568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:20.506693 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:20.506996 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:20.506998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:20.507371 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:20.507532 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:20.507540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.508612 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.508617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:20.509590 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:20.509602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.510240 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.510244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:20.510378 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:20.515768 (+ 5390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:20.515792 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:20.519434 (+ 3642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":239,"threads_started":1}
W20260926 19:40:20.519814 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:20.506146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:46909 (local address 127.2.164.126:40687)
0926 19:40:20.506302 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:20.506307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:20.506689 (+ 382us) server_negotiation.cc:407] Connection header received
0926 19:40:20.506805 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:20.506810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:20.506891 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:20.506969 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:20.507638 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.508485 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.509691 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.510122 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.510212 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:20.519602 (+ 9390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:46909: BlockingRecv error: recv got EOF from 127.2.164.66:46909 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 19:40:20 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:20 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:20.916290 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:20.908092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:59049)
0926 19:40:20.908550 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:20.908565 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:20.908700 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:20.909057 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:20.909062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:20.909539 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:20.909702 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:20.909710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.910663 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.910668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:20.911589 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:20.911600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.912303 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.912308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:20.912464 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:20.912983 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:20.912999 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:20.916060 (+ 3061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":241,"threads_started":1}
W20260926 19:40:20.916435 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:20.908105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:59049 (local address 127.2.164.126:40687)
0926 19:40:20.908247 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:20.908254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:20.908651 (+ 397us) server_negotiation.cc:407] Connection header received
0926 19:40:20.908855 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:20.908860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:20.908934 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:20.909010 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:20.909843 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.910527 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.911719 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:20.912138 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:20.912239 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:20.916238 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:59049: BlockingRecv error: recv got EOF from 127.2.164.65:59049 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 19:40:21 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:21 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:21.050683 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:21.042406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:35087)
0926 19:40:21.042922 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:21.042941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:21.043083 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:21.043429 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:21.043433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:21.043823 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:21.044007 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:21.044016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.044991 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.044996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:21.045951 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:21.045963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.046636 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.046642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:21.046813 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:21.047365 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:21.047382 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:21.050467 (+ 3085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":310,"threads_started":1}
W20260926 19:40:21.050890 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:21.042475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:35087 (local address 127.2.164.126:40687)
0926 19:40:21.042655 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:21.042662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:21.043022 (+ 360us) server_negotiation.cc:407] Connection header received
0926 19:40:21.043200 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:21.043204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:21.043296 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:21.043374 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:21.044138 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.044858 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.046078 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.046494 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.046604 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:21.050637 (+ 4033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:35087: BlockingRecv error: recv got EOF from 127.2.164.67:35087 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 19:40:21 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:21 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:21.529196 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:21.520477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:46547)
0926 19:40:21.521013 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:21.521028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:21.521151 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:21.521492 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:21.521497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:21.521889 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:21.522046 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:21.522054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.523004 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.523010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:21.523927 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:21.523938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.524651 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.524657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:21.524825 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:21.525476 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:21.525494 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:21.528878 (+ 3384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":280,"threads_started":1}
W20260926 19:40:21.529343 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:21.520592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:46547 (local address 127.2.164.126:40687)
0926 19:40:21.520805 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:21.520811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:21.521105 (+ 294us) server_negotiation.cc:407] Connection header received
0926 19:40:21.521295 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:21.521299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:21.521373 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:21.521456 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:21.522180 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.522874 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.524059 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.524493 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.524596 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:21.529072 (+ 4476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:46547: BlockingRecv error: recv got EOF from 127.2.164.66:46547 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 19:40:21 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:21 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:21.926162 11952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:21.917220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:56997)
0926 19:40:21.917651 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:21.917668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:21.917786 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:21.918099 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:21.918103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:21.918503 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:21.918657 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:21.918666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.919714 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.919717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:21.920685 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:21.920703 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.921632 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.921637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:21.921807 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:21.922328 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:21.922344 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:21.925870 (+ 3526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":237,"threads_started":1}
W20260926 19:40:21.926288 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:21.917303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:56997 (local address 127.2.164.126:40687)
0926 19:40:21.917494 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:21.917500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:21.917738 (+ 238us) server_negotiation.cc:407] Connection header received
0926 19:40:21.917869 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:21.917873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:21.917941 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:21.918016 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:21.918784 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.919589 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.920830 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:21.921488 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:21.921620 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:21.926070 (+ 4450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:56997: BlockingRecv error: recv got EOF from 127.2.164.65:56997 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 19:40:22 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:22 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:22.059779 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:22.051494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:54093)
0926 19:40:22.052009 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:22.052024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:22.052135 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:22.052412 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:22.052416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:22.052855 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:22.053019 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:22.053028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.054159 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.054164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:22.055085 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:22.055098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.055783 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.055788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:22.055914 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:22.056446 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:22.056464 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:22.059556 (+ 3092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":305,"threads_started":1}
W20260926 19:40:22.059994 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:22.051589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:54093 (local address 127.2.164.126:40687)
0926 19:40:22.051760 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:22.051764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:22.052086 (+ 322us) server_negotiation.cc:407] Connection header received
0926 19:40:22.052233 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:22.052237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:22.052297 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:22.052377 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:22.053159 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.054030 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.055206 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.055616 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.055716 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:22.059767 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:54093: BlockingRecv error: recv got EOF from 127.2.164.67:54093 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 19:40:22 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:22 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:22.538493 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:22.530146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:45819)
0926 19:40:22.530593 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:22.530608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:22.530723 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:22.531009 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:22.531012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:22.531385 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:22.531537 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:22.531545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.532484 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.532487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:22.533547 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:22.533560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.534474 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.534478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:22.534595 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:22.535115 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:22.535132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:22.538285 (+ 3153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":242,"threads_started":1}
W20260926 19:40:22.538671 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:22.530329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:45819 (local address 127.2.164.126:40687)
0926 19:40:22.530480 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:22.530484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:22.530696 (+ 212us) server_negotiation.cc:407] Connection header received
0926 19:40:22.530819 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:22.530822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:22.530898 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:22.530983 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:22.531667 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.532371 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.533682 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.534346 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.534450 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:22.538452 (+ 4002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:45819: BlockingRecv error: recv got EOF from 127.2.164.66:45819 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 19:40:22 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:22 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.65@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:22.935312 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:22.927333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.65:40871)
0926 19:40:22.927810 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:22.927824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:22.927944 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:22.928288 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:22.928292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:22.928701 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:22.928905 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:22.928915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.929864 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.929868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:22.930741 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:22.930750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.931357 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.931361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:22.931476 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:22.932016 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:22.932034 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:22.935077 (+ 3043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":239,"threads_started":1}
W20260926 19:40:22.935480 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:22.927448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:40871 (local address 127.2.164.126:40687)
0926 19:40:22.927643 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:22.927650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:22.927911 (+ 261us) server_negotiation.cc:407] Connection header received
0926 19:40:22.928060 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:22.928065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:22.928142 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:22.928240 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:22.929027 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.929748 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.930860 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:22.931259 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:22.931347 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:22.935262 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:40871: BlockingRecv error: recv got EOF from 127.2.164.65:40871 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 19:40:23 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:23 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.67@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:23.068754 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:23.060610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.67:44241)
0926 19:40:23.061053 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:23.061069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:23.061200 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:23.061527 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:23.061531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:23.061918 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:23.062068 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:23.062076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.063059 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.063063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:23.063974 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:23.063984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.064601 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.064604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:23.064715 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:23.065287 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:23.065304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:23.068473 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":238,"threads_started":1}
W20260926 19:40:23.069104 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:23.060686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:44241 (local address 127.2.164.126:40687)
0926 19:40:23.060916 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:23.060920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:23.061129 (+ 209us) server_negotiation.cc:407] Connection header received
0926 19:40:23.061302 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:23.061307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:23.061398 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:23.061480 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:23.062182 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.062913 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.064085 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.064499 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.064587 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:23.068721 (+ 4134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.67:44241: BlockingRecv error: recv got EOF from 127.2.164.67:44241 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 26 19:40:23 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:40:23 dist-test-slave-mrw4 krb5kdc[10911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.164.66@KRBTEST.COM for oryx/127.2.164.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:40:23.547653 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:23.539496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40687 (local address 127.2.164.66:33139)
0926 19:40:23.539963 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:23.539978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:23.540082 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:23.540343 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:23.540346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:23.540682 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:23.540869 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:23.540878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.541973 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.541978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:23.542986 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:23.542999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.543688 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.543694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:23.543851 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:23.544408 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:23.544425 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:23.547418 (+ 2993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.164.126:40687: Server oryx/127.2.164.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":213,"threads_started":1}
W20260926 19:40:23.547791 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:23.539584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:33139 (local address 127.2.164.126:40687)
0926 19:40:23.539732 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:23.539736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:23.540039 (+ 303us) server_negotiation.cc:407] Connection header received
0926 19:40:23.540166 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:23.540169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:23.540240 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:23.540319 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:23.540972 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.541798 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.543117 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.543548 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.543649 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:23.547598 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.66:33139: BlockingRecv error: recv got EOF from 127.2.164.66:33139 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:40:23.924111 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 11498
I20260926 19:40:23.935957 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:23.925080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.2.164.65:36941)
0926 19:40:23.925715 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:23.925722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:23.925749 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:40:23.925812 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:23.925818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:23.926075 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:23.926250 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:23.927469 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.928522 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.929691 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.930267 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.930375 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:23.932625 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:23.932634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:23.932648 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:23.932695 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:23.935392 (+ 2697us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.164.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 19:40:23.935662 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52314: Request ticket server kudu/127.2.164.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":346,"threads_started":1}
W20260926 19:40:23.936045 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 11498: Not authorized: Client connection negotiation failed: client connection to 127.2.164.65:36941: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.164.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 19:40:23.936245 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 11498
W20260926 19:40:23.950075 11647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:23.936853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:41757 (local address 127.2.164.126:40687)
0926 19:40:23.937147 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:23.937153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:23.949866 (+ 12713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.164.65:41757: BlockingRecv error: recv got EOF from 127.2.164.65:41757 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
I20260926 19:40:23.950152 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 11649
I20260926 19:40:23.961493 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:23.951142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.2.164.66:34901)
0926 19:40:23.951721 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:23.951728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:23.951756 (+ 28us) server_negotiation.cc:407] Connection header received
0926 19:40:23.951815 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:23.951821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:23.952059 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:23.952218 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:23.953773 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.954819 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.955933 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.956478 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.956568 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:23.958712 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:23.958720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:23.958734 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:23.958771 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:23.960959 (+ 2188us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.164.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 19:40:23.961261 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49880: Request ticket server kudu/127.2.164.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":285,"threads_started":1}
W20260926 19:40:23.961558 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 11649: Not authorized: Client connection negotiation failed: client connection to 127.2.164.66:34901: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.164.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 19:40:23.961584 11961 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49880: Request ticket server kudu/127.2.164.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 19:40:23.961668 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 11649
I20260926 19:40:23.974668 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 11782
I20260926 19:40:23.987771 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:23.975713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.2.164.67:32997)
0926 19:40:23.977536 (+ 1823us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:23.977546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:23.977576 (+ 30us) server_negotiation.cc:407] Connection header received
0926 19:40:23.977704 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:23.977715 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:23.978064 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:23.978299 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:23.979741 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.980850 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.982073 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:23.982594 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:23.982687 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:23.984886 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:23.984894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:23.984907 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:23.984945 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:23.987279 (+ 2334us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.164.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 19:40:23.987548 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38088: Request ticket server kudu/127.2.164.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":1519,"thread_start_us":1306,"threads_started":1}
W20260926 19:40:23.987859 11962 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38088: Request ticket server kudu/127.2.164.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 19:40:23.987864 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 11782: Not authorized: Client connection negotiation failed: client connection to 127.2.164.67:32997: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.164.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 19:40:23.988015 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 11782
I20260926 19:40:24.000710 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 11429
I20260926 19:40:24.001899 11491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:24.113081 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 11429
2026-09-26T19:40:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20802 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T19:40:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:40:24Z Disabled control of system clock
I20260926 19:40:24.178087 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36305
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:36305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 19:40:24.485286 11967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:24.485548 11967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:24.485662 11967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:24.492547 11967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:40:24.492635 11967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:24.492690 11967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:24.492769 11967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:40:24.492826 11967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:40:24.501493 11967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:36305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:24.502739 11967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:24.504558 11967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:24.515902 11972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:24.515832 11973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:24.515874 11975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:24.515959 11967 server_base.cc:1034] running on GCE node
I20260926 19:40:24.516985 11967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:24.518283 11967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:24.519483 11967 hybrid_clock.cc:648] HybridClock initialized: now 1790451624519434 us; error 44 us; skew 500 ppm
I20260926 19:40:24.519790 11967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:40:24.521863 11967 webserver.cc:460] Webserver started at http://127.2.164.126:35301/ using document root <none> and password file <none>
I20260926 19:40:24.522508 11967 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:24.522600 11967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:24.522905 11967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:24.525660 11967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "584497c3fa97455cb28938aee6133ad3"
format_stamp: "Formatted at 2026-09-26 19:40:24 on dist-test-slave-mrw4"
server_key: "2a8b555f6768f38f1ed1506f4fa71138"
server_key_iv: "05f794900e2ab4b66fa6be3e01fec274"
server_key_version: "encryptionkey@0"
I20260926 19:40:24.526381 11967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "584497c3fa97455cb28938aee6133ad3"
format_stamp: "Formatted at 2026-09-26 19:40:24 on dist-test-slave-mrw4"
server_key: "2a8b555f6768f38f1ed1506f4fa71138"
server_key_iv: "05f794900e2ab4b66fa6be3e01fec274"
server_key_version: "encryptionkey@0"
I20260926 19:40:24.531005 11967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:40:24.534090 11981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:24.535274 11967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 19:40:24.535434 11967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "584497c3fa97455cb28938aee6133ad3"
format_stamp: "Formatted at 2026-09-26 19:40:24 on dist-test-slave-mrw4"
server_key: "2a8b555f6768f38f1ed1506f4fa71138"
server_key_iv: "05f794900e2ab4b66fa6be3e01fec274"
server_key_version: "encryptionkey@0"
I20260926 19:40:24.535606 11967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:24.552973 11967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:24.553726 11967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:24.553978 11967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:24.569149 11967 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:36305
I20260926 19:40:24.569187 12032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:36305 every 8 connection(s)
I20260926 19:40:24.570392 11967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:40:24.573899 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:24.574458 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 11967
I20260926 19:40:24.574607 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:40:24.575241 2705 external_mini_cluster.cc:1428] Setting key 00a17f754d42d9a534fb7a45658d3b12
I20260926 19:40:24.587211 12033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3: Bootstrap starting.
I20260926 19:40:24.589452 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:24.577050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.2.164.126:36305)
0926 19:40:24.578133 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:24.578148 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:24.578204 (+ 56us) server_negotiation.cc:407] Connection header received
0926 19:40:24.579988 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:24.580015 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:24.580593 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:24.581022 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:24.581536 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:24.583383 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:24.584613 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:24.585631 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:24.585792 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:24.586756 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:24.586769 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:24.586786 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:24.587456 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:24.587554 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:24.587589 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:24.587699 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:24.587864 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 19:40:24.588160 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":315,"threads_started":1}
I20260926 19:40:24.591013 12033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:24.592038 12033 log.cc:824] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:40:24.594478 12033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3: No bootstrap required, opened a new log
I20260926 19:40:24.601296 12033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584497c3fa97455cb28938aee6133ad3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36305 } }
I20260926 19:40:24.601608 12033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:24.601696 12033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584497c3fa97455cb28938aee6133ad3, State: Initialized, Role: FOLLOWER
I20260926 19:40:24.602217 12033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584497c3fa97455cb28938aee6133ad3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36305 } }
I20260926 19:40:24.602416 12033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:40:24.602535 12033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:40:24.602666 12033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:24.604588 12033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584497c3fa97455cb28938aee6133ad3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36305 } }
I20260926 19:40:24.605031 12033 leader_election.cc:302] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 584497c3fa97455cb28938aee6133ad3; no voters:
I20260926 19:40:24.605548 12033 leader_election.cc:288] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:40:24.605656 12038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:40:24.606283 12038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [term 1 LEADER]: Becoming Leader. State: Replica: 584497c3fa97455cb28938aee6133ad3, State: Running, Role: LEADER
I20260926 19:40:24.606750 12038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584497c3fa97455cb28938aee6133ad3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36305 } }
I20260926 19:40:24.607275 12033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:40:24.609141 12039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "584497c3fa97455cb28938aee6133ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584497c3fa97455cb28938aee6133ad3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36305 } } }
I20260926 19:40:24.609387 12040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 584497c3fa97455cb28938aee6133ad3. Latest consensus state: current_term: 1 leader_uuid: "584497c3fa97455cb28938aee6133ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584497c3fa97455cb28938aee6133ad3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36305 } } }
I20260926 19:40:24.609655 12039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:40:24.609711 12040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:40:24.611526 12047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:40:24.615763 12047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:40:24.622942 12047 catalog_manager.cc:1347] Generated new cluster ID: ef3b7000809345b390d1baa05bf9d63d
I20260926 19:40:24.623027 12047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:40:24.647867 12047 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:40:24.649003 12047 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:40:24.656989 12047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 584497c3fa97455cb28938aee6133ad3: Generated new TSK 0
I20260926 19:40:24.657718 12047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:40:24.661777 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36305
--builtin_ntp_servers=127.2.164.84:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 19:40:24.963482 12057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:24.963727 12057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:24.963847 12057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:24.970813 12057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:24.970898 12057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:24.971019 12057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:40:24.979671 12057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:24.981071 12057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:24.983294 12057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:24.995393 12062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:24.995424 12065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:24.995451 12063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:24.996194 12057 server_base.cc:1034] running on GCE node
I20260926 19:40:24.996763 12057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:24.997656 12057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:24.998850 12057 hybrid_clock.cc:648] HybridClock initialized: now 1790451624998796 us; error 51 us; skew 500 ppm
I20260926 19:40:24.999130 12057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:40:25.001093 12057 webserver.cc:460] Webserver started at http://127.2.164.65:43829/ using document root <none> and password file <none>
I20260926 19:40:25.001751 12057 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:25.001853 12057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:25.002137 12057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:25.004685 12057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "792d651d37d0475983fa3ed48e2b78c3"
format_stamp: "Formatted at 2026-09-26 19:40:25 on dist-test-slave-mrw4"
server_key: "f62c42cb121f2c45d38e2943eeadd49a"
server_key_iv: "48bd930282a94cb5b87722e16b29da3a"
server_key_version: "encryptionkey@0"
I20260926 19:40:25.005446 12057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "792d651d37d0475983fa3ed48e2b78c3"
format_stamp: "Formatted at 2026-09-26 19:40:25 on dist-test-slave-mrw4"
server_key: "f62c42cb121f2c45d38e2943eeadd49a"
server_key_iv: "48bd930282a94cb5b87722e16b29da3a"
server_key_version: "encryptionkey@0"
I20260926 19:40:25.010278 12057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:40:25.013420 12071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:25.014618 12057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:40:25.014758 12057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "792d651d37d0475983fa3ed48e2b78c3"
format_stamp: "Formatted at 2026-09-26 19:40:25 on dist-test-slave-mrw4"
server_key: "f62c42cb121f2c45d38e2943eeadd49a"
server_key_iv: "48bd930282a94cb5b87722e16b29da3a"
server_key_version: "encryptionkey@0"
I20260926 19:40:25.014945 12057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:25.039685 12057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:25.040458 12057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:25.040719 12057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:25.041615 12057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:25.042941 12057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:25.043030 12057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:25.043129 12057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:25.043195 12057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:25.071193 12057 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:39353
I20260926 19:40:25.071259 12183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:39353 every 8 connection(s)
I20260926 19:40:25.072404 12057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:40:25.080149 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 12057
I20260926 19:40:25.080332 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:40:25.080955 2705 external_mini_cluster.cc:1428] Setting key dc0668e13835066ff9a40369c487feb0
I20260926 19:40:25.083688 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36305
--builtin_ntp_servers=127.2.164.84:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 19:40:25.084223 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:25.074839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:59819 (local address 127.2.164.126:36305)
0926 19:40:25.075008 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:25.075013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:25.076386 (+ 1373us) server_negotiation.cc:407] Connection header received
0926 19:40:25.078117 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:25.078122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:25.078202 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:25.078284 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:25.079315 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.080000 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.081614 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.082092 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.082211 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:25.083478 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:25.083484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:25.083489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:25.083510 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:25.083548 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:25.083564 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:25.083653 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:25.083843 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 19:40:25.083933 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:40:25.085147 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:25.075377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36305 (local address 127.2.164.65:59819)
0926 19:40:25.076232 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:25.076276 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:25.077852 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:25.078508 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:25.078516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:25.078607 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:25.079152 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:25.079168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.080143 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.080146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:25.081484 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:25.081496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.082281 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.082286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:25.082515 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:25.083162 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:40:25.083164 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:25.083202 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:40:25.083277 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:25.083645 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:25.083651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:25.083656 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:40:25.083794 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 19:40:25.084163 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":240,"threads_started":1}
I20260926 19:40:25.086475 12184 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36305
I20260926 19:40:25.086725 12184 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:25.087393 12184 heartbeater.cc:511] Master 127.2.164.126:36305 requested a full tablet report, sending...
I20260926 19:40:25.089417 11998 ts_manager.cc:194] Registered new tserver with Master: 792d651d37d0475983fa3ed48e2b78c3 (127.2.164.65:39353)
I20260926 19:40:25.090960 11998 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.65:59819
W20260926 19:40:25.383428 12187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:25.383692 12187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:25.383812 12187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:25.390756 12187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:25.390861 12187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:25.391002 12187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:40:25.399664 12187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:25.400831 12187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:25.402500 12187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:25.414264 12195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:25.414324 12192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:25.414322 12193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:25.415163 12187 server_base.cc:1034] running on GCE node
I20260926 19:40:25.415691 12187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:25.416553 12187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:25.417737 12187 hybrid_clock.cc:648] HybridClock initialized: now 1790451625417676 us; error 53 us; skew 500 ppm
I20260926 19:40:25.418015 12187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:40:25.420046 12187 webserver.cc:460] Webserver started at http://127.2.164.66:36025/ using document root <none> and password file <none>
I20260926 19:40:25.420660 12187 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:25.420745 12187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:25.421031 12187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:25.423611 12187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "9236205c83604366b8c1ebf532b817f3"
format_stamp: "Formatted at 2026-09-26 19:40:25 on dist-test-slave-mrw4"
server_key: "84467e82921002f96bfcbfe76c4c3150"
server_key_iv: "4d6702cab540934ea1a19ee67fa81bd9"
server_key_version: "encryptionkey@0"
I20260926 19:40:25.424314 12187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "9236205c83604366b8c1ebf532b817f3"
format_stamp: "Formatted at 2026-09-26 19:40:25 on dist-test-slave-mrw4"
server_key: "84467e82921002f96bfcbfe76c4c3150"
server_key_iv: "4d6702cab540934ea1a19ee67fa81bd9"
server_key_version: "encryptionkey@0"
I20260926 19:40:25.428840 12187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260926 19:40:25.431897 12201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:25.433061 12187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:40:25.433213 12187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "9236205c83604366b8c1ebf532b817f3"
format_stamp: "Formatted at 2026-09-26 19:40:25 on dist-test-slave-mrw4"
server_key: "84467e82921002f96bfcbfe76c4c3150"
server_key_iv: "4d6702cab540934ea1a19ee67fa81bd9"
server_key_version: "encryptionkey@0"
I20260926 19:40:25.433389 12187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:25.450551 12187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:25.451494 12187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:25.451803 12187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:25.452869 12187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:25.454474 12187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:25.454610 12187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 19:40:25.454720 12187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:25.454800 12187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:25.483672 12187 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:45889
I20260926 19:40:25.483717 12313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:45889 every 8 connection(s)
I20260926 19:40:25.484880 12187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:40:25.492362 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 12187
I20260926 19:40:25.492491 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:40:25.492885 2705 external_mini_cluster.cc:1428] Setting key ae6c54a8b83a28d341d695cd46661b7a
I20260926 19:40:25.495399 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36305
--builtin_ntp_servers=127.2.164.84:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 19:40:25.496520 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:25.487217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:51015 (local address 127.2.164.126:36305)
0926 19:40:25.487368 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:25.487372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:25.488696 (+ 1324us) server_negotiation.cc:407] Connection header received
0926 19:40:25.490427 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:25.490432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:25.490507 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:25.490588 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:25.491606 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.492330 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.493967 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.494412 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.494519 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:25.495810 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:25.495816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:25.495820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:25.495841 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:25.495877 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:25.495890 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:25.495980 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:25.496133 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 19:40:25.496220 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 19:40:25.497388 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:25.487685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36305 (local address 127.2.164.66:51015)
0926 19:40:25.488534 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:25.488573 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:25.490161 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:25.490815 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:25.490824 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:25.490911 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:25.491445 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:25.491462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.492477 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.492482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:25.493834 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:25.493855 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.494616 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.494624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:25.494852 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:25.495480 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:40:25.495483 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:25.495539 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:40:25.495629 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:25.495974 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:25.495980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:25.495984 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 19:40:25.496108 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 19:40:25.496382 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":232,"threads_started":1}
I20260926 19:40:25.501745 12314 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36305
I20260926 19:40:25.502058 12314 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:25.502903 12314 heartbeater.cc:511] Master 127.2.164.126:36305 requested a full tablet report, sending...
I20260926 19:40:25.504323 11998 ts_manager.cc:194] Registered new tserver with Master: 9236205c83604366b8c1ebf532b817f3 (127.2.164.66:45889)
I20260926 19:40:25.505347 11998 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.66:51015
W20260926 19:40:25.812757 12317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:25.813002 12317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:25.813090 12317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:25.820063 12317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:25.820151 12317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:25.820258 12317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:40:25.828856 12317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:25.830057 12317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:25.831691 12317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:25.843528 12322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:25.843602 12323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:25.843572 12325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:25.844377 12317 server_base.cc:1034] running on GCE node
I20260926 19:40:25.844920 12317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:25.845774 12317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:25.846966 12317 hybrid_clock.cc:648] HybridClock initialized: now 1790451625846927 us; error 27 us; skew 500 ppm
I20260926 19:40:25.847249 12317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:40:25.849267 12317 webserver.cc:460] Webserver started at http://127.2.164.67:35543/ using document root <none> and password file <none>
I20260926 19:40:25.849861 12317 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:25.849951 12317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:25.850241 12317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:25.852792 12317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "dd05acb5d58641489e4332e5a4e71287"
format_stamp: "Formatted at 2026-09-26 19:40:25 on dist-test-slave-mrw4"
server_key: "f04d5fac93bce8bc3b986653463723d5"
server_key_iv: "7ffd9166ebb8121a9621ea2f482bfa36"
server_key_version: "encryptionkey@0"
I20260926 19:40:25.853539 12317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "dd05acb5d58641489e4332e5a4e71287"
format_stamp: "Formatted at 2026-09-26 19:40:25 on dist-test-slave-mrw4"
server_key: "f04d5fac93bce8bc3b986653463723d5"
server_key_iv: "7ffd9166ebb8121a9621ea2f482bfa36"
server_key_version: "encryptionkey@0"
I20260926 19:40:25.858068 12317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:40:25.861135 12331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:25.862350 12317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:40:25.862488 12317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "dd05acb5d58641489e4332e5a4e71287"
format_stamp: "Formatted at 2026-09-26 19:40:25 on dist-test-slave-mrw4"
server_key: "f04d5fac93bce8bc3b986653463723d5"
server_key_iv: "7ffd9166ebb8121a9621ea2f482bfa36"
server_key_version: "encryptionkey@0"
I20260926 19:40:25.862668 12317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:25.876480 12317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:25.877238 12317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:25.877483 12317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:25.878268 12317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:25.879642 12317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:25.879732 12317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:25.879829 12317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:25.879894 12317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:25.907681 12317 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:37405
I20260926 19:40:25.907743 12443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:37405 every 8 connection(s)
I20260926 19:40:25.908916 12317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:40:25.915881 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 12317
I20260926 19:40:25.916013 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:40:25.916429 2705 external_mini_cluster.cc:1428] Setting key da677586b996c29611b24c796c1d09ff
I20260926 19:40:25.920481 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:25.911249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:56311 (local address 127.2.164.126:36305)
0926 19:40:25.911427 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:25.911433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:25.912737 (+ 1304us) server_negotiation.cc:407] Connection header received
0926 19:40:25.914421 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:25.914426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:25.914499 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:25.914581 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:25.915580 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.916322 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.918055 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.918459 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.918565 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:25.919827 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:25.919832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:25.919835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:25.919850 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:25.919878 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:25.919888 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:25.919950 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:25.920191 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 19:40:25.920257 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:40:25.921558 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:25.911774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36305 (local address 127.2.164.67:56311)
0926 19:40:25.912592 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:25.912634 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:25.914178 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:25.914811 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:25.914819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:25.914902 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:25.915429 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:25.915445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.916452 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.916457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:25.917931 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:25.917943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.918659 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.918666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:25.918903 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:25.919543 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:40:25.919546 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:25.919583 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:40:25.919659 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:25.919995 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:25.920004 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:25.920011 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 19:40:25.920187 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 19:40:25.920466 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":238,"threads_started":1}
I20260926 19:40:25.922734 12444 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36305
I20260926 19:40:25.922988 12444 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:25.923684 12444 heartbeater.cc:511] Master 127.2.164.126:36305 requested a full tablet report, sending...
I20260926 19:40:25.924949 11998 ts_manager.cc:194] Registered new tserver with Master: dd05acb5d58641489e4332e5a4e71287 (127.2.164.67:37405)
I20260926 19:40:25.925737 11998 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.67:56311
I20260926 19:40:25.931142 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 19:40:25.936846 12036 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:25.935362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.2.164.126:36305)
0926 19:40:25.935512 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:25.935518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:25.935908 (+ 390us) server_negotiation.cc:407] Connection header received
0926 19:40:25.936115 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:25.936119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:25.936183 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:25.936259 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:25.936606 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35572: BlockingRecv error: recv got EOF from 127.0.0.1:35572 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 19:40:25.937237 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 12057
I20260926 19:40:25.944195 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:25.938089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.2.164.65:39353)
0926 19:40:25.938580 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:25.938586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:25.938610 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:40:25.938670 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:25.938675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:25.938854 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:25.938965 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:25.939304 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.940309 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.941527 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:25.942065 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:25.942159 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:25.942994 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:25.943001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:25.943012 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:25.943544 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:25.943587 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:25.943604 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:25.943728 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:25.943778 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 19:40:25.943881 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":243,"threads_started":1}
I20260926 19:40:25.945382 12179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:26.025662 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 12057
I20260926 19:40:26.039719 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 12187
I20260926 19:40:26.047021 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:26.040512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.2.164.66:45889)
0926 19:40:26.041020 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:26.041027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:26.041052 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:40:26.041110 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:26.041116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:26.041395 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:26.041524 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:26.041927 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:26.042891 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:26.044075 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:26.044596 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:26.044683 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:26.045700 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:26.045708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:26.045721 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:26.046303 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:26.046347 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:26.046365 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:26.046451 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:26.046604 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 19:40:26.046727 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260926 19:40:26.048163 12309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:26.126636 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 12187
I20260926 19:40:26.139400 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 12317
I20260926 19:40:26.146911 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:26.140201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.2.164.67:37405)
0926 19:40:26.140724 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:26.140731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:26.140760 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:40:26.140821 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:26.140827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:26.141038 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:26.141198 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:26.141627 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:26.142631 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:26.143847 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:26.144392 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:26.144506 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:26.145598 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:26.145606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:26.145619 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:26.146178 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:26.146223 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:26.146241 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:26.146355 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:26.146477 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 19:40:26.146597 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":240,"threads_started":1}
I20260926 19:40:26.148089 12439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:26.227540 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 12317
I20260926 19:40:26.241856 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 11967
I20260926 19:40:26.243028 12028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:26.341468 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 11967
2026-09-26T19:40:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2223 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T19:40:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:40:26Z Disabled control of system clock
I20260926 19:40:26.403259 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33311
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:33311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 19:40:26.705129 12463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:26.705451 12463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:26.705556 12463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:26.712603 12463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:40:26.712697 12463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:26.712791 12463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:26.712862 12463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:40:26.712922 12463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:40:26.721829 12463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:33311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:26.723001 12463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:26.724681 12463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:26.735350 12468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:26.735366 12471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:26.735375 12469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:26.736326 12463 server_base.cc:1034] running on GCE node
I20260926 19:40:26.737073 12463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:26.738395 12463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:26.739583 12463 hybrid_clock.cc:648] HybridClock initialized: now 1790451626739536 us; error 45 us; skew 500 ppm
I20260926 19:40:26.739890 12463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:40:26.741997 12463 webserver.cc:460] Webserver started at http://127.2.164.126:42359/ using document root <none> and password file <none>
I20260926 19:40:26.742619 12463 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:26.742710 12463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:26.743002 12463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:26.745671 12463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "bea53c462b714f35aca01d080d5397f2"
format_stamp: "Formatted at 2026-09-26 19:40:26 on dist-test-slave-mrw4"
server_key: "e64017ce304b65de5967e27e16295d58"
server_key_iv: "c1d216ce6e73dbe66f3c06dcbb2c2b90"
server_key_version: "encryptionkey@0"
I20260926 19:40:26.746387 12463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "bea53c462b714f35aca01d080d5397f2"
format_stamp: "Formatted at 2026-09-26 19:40:26 on dist-test-slave-mrw4"
server_key: "e64017ce304b65de5967e27e16295d58"
server_key_iv: "c1d216ce6e73dbe66f3c06dcbb2c2b90"
server_key_version: "encryptionkey@0"
I20260926 19:40:26.751034 12463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260926 19:40:26.754110 12477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:26.755337 12463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:40:26.755475 12463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "bea53c462b714f35aca01d080d5397f2"
format_stamp: "Formatted at 2026-09-26 19:40:26 on dist-test-slave-mrw4"
server_key: "e64017ce304b65de5967e27e16295d58"
server_key_iv: "c1d216ce6e73dbe66f3c06dcbb2c2b90"
server_key_version: "encryptionkey@0"
I20260926 19:40:26.755646 12463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:26.761040 12463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:26.761729 12463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:26.761981 12463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:26.777947 12463 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:33311
I20260926 19:40:26.777978 12528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:33311 every 8 connection(s)
I20260926 19:40:26.779116 12463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:40:26.779695 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 12463
I20260926 19:40:26.779886 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:40:26.780572 2705 external_mini_cluster.cc:1428] Setting key cc6a3de41a614ff4734dc8543c037772
I20260926 19:40:26.783135 12529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:26.788139 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:26.782420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.2.164.126:33311)
0926 19:40:26.783369 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:26.783381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:26.783420 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:40:26.784888 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:26.784908 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:26.785333 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:26.785683 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:26.786626 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:26.786636 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:26.786648 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:26.786662 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:26.786736 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:26.786751 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:26.786954 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:26.787084 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 19:40:26.787310 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":329,"threads_started":1}
I20260926 19:40:26.793465 12529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2: Bootstrap starting.
I20260926 19:40:26.796365 12529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:26.797302 12529 log.cc:824] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:40:26.799726 12529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2: No bootstrap required, opened a new log
I20260926 19:40:26.806653 12529 raft_consensus.cc:348] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea53c462b714f35aca01d080d5397f2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33311 } }
I20260926 19:40:26.806946 12529 raft_consensus.cc:374] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:26.807029 12529 raft_consensus.cc:729] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea53c462b714f35aca01d080d5397f2, State: Initialized, Role: FOLLOWER
I20260926 19:40:26.807466 12529 consensus_queue.cc:260] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea53c462b714f35aca01d080d5397f2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33311 } }
I20260926 19:40:26.807646 12529 raft_consensus.cc:388] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:40:26.807731 12529 raft_consensus.cc:482] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:40:26.807844 12529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:26.809787 12529 raft_consensus.cc:504] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea53c462b714f35aca01d080d5397f2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33311 } }
I20260926 19:40:26.810184 12529 leader_election.cc:302] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bea53c462b714f35aca01d080d5397f2; no voters:
I20260926 19:40:26.810655 12529 leader_election.cc:288] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:40:26.810740 12534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:40:26.811317 12534 raft_consensus.cc:686] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [term 1 LEADER]: Becoming Leader. State: Replica: bea53c462b714f35aca01d080d5397f2, State: Running, Role: LEADER
I20260926 19:40:26.811811 12534 consensus_queue.cc:237] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea53c462b714f35aca01d080d5397f2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33311 } }
I20260926 19:40:26.812449 12529 sys_catalog.cc:564] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:40:26.814450 12535 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bea53c462b714f35aca01d080d5397f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea53c462b714f35aca01d080d5397f2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33311 } } }
I20260926 19:40:26.814445 12536 sys_catalog.cc:455] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bea53c462b714f35aca01d080d5397f2. Latest consensus state: current_term: 1 leader_uuid: "bea53c462b714f35aca01d080d5397f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bea53c462b714f35aca01d080d5397f2" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33311 } } }
I20260926 19:40:26.814822 12535 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [sys.catalog]: This master's current role is: LEADER
I20260926 19:40:26.815614 12536 sys_catalog.cc:458] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2 [sys.catalog]: This master's current role is: LEADER
I20260926 19:40:26.816543 12542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:40:26.820995 12542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:40:26.828938 12542 catalog_manager.cc:1347] Generated new cluster ID: 28b1cca9bedf4528a361b2c41f6fd6d5
I20260926 19:40:26.829037 12542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:40:26.845831 12542 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:40:26.846277 12542 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:40:26.855829 12542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bea53c462b714f35aca01d080d5397f2: Generated new TSK 0
I20260926 19:40:26.856879 12542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:40:26.872386 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33311
--builtin_ntp_servers=127.2.164.84:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 19:40:27.184427 12553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:27.184706 12553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:27.184820 12553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:27.192098 12553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:27.192203 12553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:27.192358 12553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:40:27.201414 12553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:27.202720 12553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:27.204602 12553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:27.217556 12558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:27.217523 12561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:27.217495 12559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:27.218247 12553 server_base.cc:1034] running on GCE node
I20260926 19:40:27.218874 12553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:27.219771 12553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:27.220968 12553 hybrid_clock.cc:648] HybridClock initialized: now 1790451627220924 us; error 32 us; skew 500 ppm
I20260926 19:40:27.221314 12553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:40:27.223469 12553 webserver.cc:460] Webserver started at http://127.2.164.65:33583/ using document root <none> and password file <none>
I20260926 19:40:27.224133 12553 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:27.224231 12553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:27.224542 12553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:27.227342 12553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "3e30c9091e2042cd97c1ad4205606759"
format_stamp: "Formatted at 2026-09-26 19:40:27 on dist-test-slave-mrw4"
server_key: "841a72708f04a7eed9072650855b6dce"
server_key_iv: "71f1d499b6c8f9dbfb09dcb67b06b9ff"
server_key_version: "encryptionkey@0"
I20260926 19:40:27.228116 12553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "3e30c9091e2042cd97c1ad4205606759"
format_stamp: "Formatted at 2026-09-26 19:40:27 on dist-test-slave-mrw4"
server_key: "841a72708f04a7eed9072650855b6dce"
server_key_iv: "71f1d499b6c8f9dbfb09dcb67b06b9ff"
server_key_version: "encryptionkey@0"
I20260926 19:40:27.232945 12553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 19:40:27.236191 12567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:27.237449 12553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 19:40:27.237588 12553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "3e30c9091e2042cd97c1ad4205606759"
format_stamp: "Formatted at 2026-09-26 19:40:27 on dist-test-slave-mrw4"
server_key: "841a72708f04a7eed9072650855b6dce"
server_key_iv: "71f1d499b6c8f9dbfb09dcb67b06b9ff"
server_key_version: "encryptionkey@0"
I20260926 19:40:27.237771 12553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:27.242990 12553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:27.243705 12553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:27.243930 12553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:27.244803 12553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:27.246150 12553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:27.246241 12553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:27.246352 12553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:27.246417 12553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:27.274591 12553 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:41581
I20260926 19:40:27.274636 12679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:41581 every 8 connection(s)
I20260926 19:40:27.275794 12553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:40:27.283053 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:27.278207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:49463 (local address 127.2.164.126:33311)
0926 19:40:27.278412 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:27.278419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:27.279734 (+ 1315us) server_negotiation.cc:407] Connection header received
0926 19:40:27.281284 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:27.281287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:27.281342 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:27.281426 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:27.282520 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:27.282524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:27.282527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:27.282544 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:27.282574 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:27.282586 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:27.282673 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:27.282803 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 19:40:27.282860 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:40:27.283684 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 12553
I20260926 19:40:27.283854 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:40:27.283938 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:27.278729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33311 (local address 127.2.164.65:49463)
0926 19:40:27.279583 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:27.279618 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:27.281039 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:27.281605 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:27.281613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:27.281698 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:27.282249 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:40:27.282251 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:27.282287 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:40:27.282378 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:27.282653 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:27.282659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:27.282663 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 19:40:27.282850 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 19:40:27.283030 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":245,"threads_started":1}
I20260926 19:40:27.284533 2705 external_mini_cluster.cc:1428] Setting key ae30585aa52e8dc4f32d0c7aaf7147e4
I20260926 19:40:27.285382 12680 heartbeater.cc:347] Connected to a master server at 127.2.164.126:33311
I20260926 19:40:27.285686 12680 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:27.286322 12680 heartbeater.cc:511] Master 127.2.164.126:33311 requested a full tablet report, sending...
I20260926 19:40:27.287528 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33311
--builtin_ntp_servers=127.2.164.84:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 19:40:27.288256 12494 ts_manager.cc:194] Registered new tserver with Master: 3e30c9091e2042cd97c1ad4205606759 (127.2.164.65:41581)
W20260926 19:40:27.594581 12683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:27.594851 12683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:27.594964 12683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:27.601899 12683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:27.601981 12683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:27.602120 12683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:40:27.610749 12683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:27.611946 12683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:27.613705 12683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:27.625751 12688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:27.625727 12691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:27.625715 12689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:27.626240 12683 server_base.cc:1034] running on GCE node
I20260926 19:40:27.626824 12683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:27.627717 12683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:27.628906 12683 hybrid_clock.cc:648] HybridClock initialized: now 1790451627628865 us; error 51 us; skew 500 ppm
I20260926 19:40:27.629304 12683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:40:27.631470 12683 webserver.cc:460] Webserver started at http://127.2.164.66:35211/ using document root <none> and password file <none>
I20260926 19:40:27.632117 12683 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:27.632225 12683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:27.632525 12683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:27.636797 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "94ceb3a481b047d6be65c59b53a506f1"
format_stamp: "Formatted at 2026-09-26 19:40:27 on dist-test-slave-mrw4"
server_key: "35f0cdea9c1c14fb2419f7c989be02b4"
server_key_iv: "6f33b557f22fd4201a4185c2a7753a5b"
server_key_version: "encryptionkey@0"
I20260926 19:40:27.637558 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "94ceb3a481b047d6be65c59b53a506f1"
format_stamp: "Formatted at 2026-09-26 19:40:27 on dist-test-slave-mrw4"
server_key: "35f0cdea9c1c14fb2419f7c989be02b4"
server_key_iv: "6f33b557f22fd4201a4185c2a7753a5b"
server_key_version: "encryptionkey@0"
I20260926 19:40:27.642467 12683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 19:40:27.645643 12697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:27.646857 12683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:40:27.647006 12683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "94ceb3a481b047d6be65c59b53a506f1"
format_stamp: "Formatted at 2026-09-26 19:40:27 on dist-test-slave-mrw4"
server_key: "35f0cdea9c1c14fb2419f7c989be02b4"
server_key_iv: "6f33b557f22fd4201a4185c2a7753a5b"
server_key_version: "encryptionkey@0"
I20260926 19:40:27.647207 12683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:27.652462 12683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:27.653237 12683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:27.653466 12683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:27.654263 12683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:27.655539 12683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:27.655628 12683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:27.655728 12683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:27.655797 12683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:27.683710 12683 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:35927
I20260926 19:40:27.683761 12809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:35927 every 8 connection(s)
I20260926 19:40:27.684921 12683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:40:27.690742 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 12683
I20260926 19:40:27.690924 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:40:27.691421 2705 external_mini_cluster.cc:1428] Setting key 1fdae7c0b6363ed10e33dde3a394289e
I20260926 19:40:27.692603 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:27.687297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:44553 (local address 127.2.164.126:33311)
0926 19:40:27.687497 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:27.687503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:27.688862 (+ 1359us) server_negotiation.cc:407] Connection header received
0926 19:40:27.690485 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:27.690489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:27.690544 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:27.690632 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:27.691987 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:27.691991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:27.691994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:27.692010 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:27.692039 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:27.692050 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:27.692151 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:27.692338 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 19:40:27.692386 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:40:27.693693 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:27.687858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33311 (local address 127.2.164.66:44553)
0926 19:40:27.688720 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:27.688755 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:27.690214 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:27.690897 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:27.690910 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:27.691031 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:27.691695 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:40:27.691698 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:27.691737 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:40:27.691818 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:27.692136 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:27.692144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:27.692150 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 19:40:27.692397 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 19:40:27.692637 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":239,"threads_started":1}
I20260926 19:40:27.694113 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33311
--builtin_ntp_servers=127.2.164.84:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 19:40:27.694924 12810 heartbeater.cc:347] Connected to a master server at 127.2.164.126:33311
I20260926 19:40:27.695221 12810 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:27.695850 12810 heartbeater.cc:511] Master 127.2.164.126:33311 requested a full tablet report, sending...
I20260926 19:40:27.697271 12494 ts_manager.cc:194] Registered new tserver with Master: 94ceb3a481b047d6be65c59b53a506f1 (127.2.164.66:35927)
W20260926 19:40:28.000593 12813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:28.000865 12813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:28.000963 12813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:28.008037 12813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:28.008122 12813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:28.008229 12813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:40:28.017192 12813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:28.018366 12813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:28.019989 12813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:28.032292 12818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:28.032253 12819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:28.032253 12821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:28.032773 12813 server_base.cc:1034] running on GCE node
I20260926 19:40:28.033468 12813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:28.034297 12813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:28.035490 12813 hybrid_clock.cc:648] HybridClock initialized: now 1790451628035441 us; error 44 us; skew 500 ppm
I20260926 19:40:28.035766 12813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:40:28.037869 12813 webserver.cc:460] Webserver started at http://127.2.164.67:38207/ using document root <none> and password file <none>
I20260926 19:40:28.038493 12813 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:28.038587 12813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:28.038892 12813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:28.041622 12813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "da2881b94d9249ef9910b79e97a7db25"
format_stamp: "Formatted at 2026-09-26 19:40:28 on dist-test-slave-mrw4"
server_key: "c158014ac69a9337b68b45d2af920720"
server_key_iv: "255a3fbcf878ce921050052b7f3afab6"
server_key_version: "encryptionkey@0"
I20260926 19:40:28.042351 12813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "da2881b94d9249ef9910b79e97a7db25"
format_stamp: "Formatted at 2026-09-26 19:40:28 on dist-test-slave-mrw4"
server_key: "c158014ac69a9337b68b45d2af920720"
server_key_iv: "255a3fbcf878ce921050052b7f3afab6"
server_key_version: "encryptionkey@0"
I20260926 19:40:28.047039 12813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:40:28.050267 12827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:28.051523 12813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 19:40:28.051659 12813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "da2881b94d9249ef9910b79e97a7db25"
format_stamp: "Formatted at 2026-09-26 19:40:28 on dist-test-slave-mrw4"
server_key: "c158014ac69a9337b68b45d2af920720"
server_key_iv: "255a3fbcf878ce921050052b7f3afab6"
server_key_version: "encryptionkey@0"
I20260926 19:40:28.051852 12813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:28.057021 12813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:28.057772 12813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:28.058012 12813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:28.058916 12813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:28.060207 12813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:28.060295 12813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:28.060396 12813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:28.060462 12813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:28.087399 12813 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:45523
I20260926 19:40:28.087460 12939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:45523 every 8 connection(s)
I20260926 19:40:28.088622 12813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:40:28.094205 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 12813
I20260926 19:40:28.094363 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:40:28.094754 2705 external_mini_cluster.cc:1428] Setting key eb722b60ecb0b91d9ca16ff885b82d0a
I20260926 19:40:28.096390 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:28.091013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:51231 (local address 127.2.164.126:33311)
0926 19:40:28.091200 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:28.091205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:28.092508 (+ 1303us) server_negotiation.cc:407] Connection header received
0926 19:40:28.094196 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:28.094202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:28.094263 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:28.094351 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:28.095699 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:28.095704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:28.095707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:28.095722 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:28.095754 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:28.095765 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:28.095885 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:28.096075 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 19:40:28.096133 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:40:28.097389 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:28.091519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33311 (local address 127.2.164.67:51231)
0926 19:40:28.092360 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:28.092395 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:28.093936 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:28.094653 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:28.094661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:28.094757 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:28.095407 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:40:28.095410 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:28.095458 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:40:28.095546 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:28.095860 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:28.095870 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:28.095878 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 19:40:28.096132 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 19:40:28.096329 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":240,"threads_started":1}
I20260926 19:40:28.098618 12940 heartbeater.cc:347] Connected to a master server at 127.2.164.126:33311
I20260926 19:40:28.098891 12940 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:28.099421 12940 heartbeater.cc:511] Master 127.2.164.126:33311 requested a full tablet report, sending...
I20260926 19:40:28.100642 12494 ts_manager.cc:194] Registered new tserver with Master: da2881b94d9249ef9910b79e97a7db25 (127.2.164.67:45523)
I20260926 19:40:28.109671 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 19:40:28.115376 12532 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:40:28.113923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.2.164.126:33311)
0926 19:40:28.114102 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:28.114110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:28.114568 (+ 458us) server_negotiation.cc:407] Connection header received
0926 19:40:28.114701 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:28.114705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:28.114754 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:28.114830 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:28.115081 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46732: BlockingRecv error: recv got EOF from 127.0.0.1:46732 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:40:28.115906 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 12553
I20260926 19:40:28.119037 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:28.116909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.2.164.65:41581)
0926 19:40:28.117393 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:28.117399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:28.117424 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:40:28.117482 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:28.117488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:28.117669 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:28.117788 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:28.118539 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:28.118546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:28.118556 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:28.118567 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:28.118606 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:28.118621 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:28.118718 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:28.118760 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 19:40:28.118838 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":195,"threads_started":1}
I20260926 19:40:28.120198 12675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:28.197948 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 12553
I20260926 19:40:28.211200 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 12683
I20260926 19:40:28.214687 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:28.212059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.2.164.66:35927)
0926 19:40:28.212584 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:28.212591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:28.212617 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:40:28.212677 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:28.212683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:28.212901 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:28.213039 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:28.214008 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:28.214015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:28.214026 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:28.214043 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:28.214082 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:28.214095 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:28.214212 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:28.214344 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 19:40:28.214446 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":256,"threads_started":1}
I20260926 19:40:28.215935 12805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:28.295183 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 12683
I20260926 19:40:28.307912 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 12813
I20260926 19:40:28.311748 12953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:28.308871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.2.164.67:45523)
0926 19:40:28.309501 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:28.309508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:28.309538 (+ 30us) server_negotiation.cc:407] Connection header received
0926 19:40:28.309594 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:28.309600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:28.309824 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:28.309959 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:28.311030 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:28.311037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:28.311049 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:40:28.311064 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:28.311100 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:40:28.311113 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:28.311258 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:28.311354 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 19:40:28.311453 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":323,"threads_started":1}
I20260926 19:40:28.312915 12935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:28.391530 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 12813
I20260926 19:40:28.403963 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 12463
I20260926 19:40:28.405055 12524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:28.501695 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 12463
2026-09-26T19:40:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2159 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:40:28 dist-test-slave-mrw4 krb5kdc[12958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:40:28 dist-test-slave-mrw4 krb5kdc[12958](info): set up 2 sockets
Sep 26 19:40:28 dist-test-slave-mrw4 krb5kdc[12958](info): commencing operation
krb5kdc: starting...
W20260926 19:40:29.509891 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:40:29 dist-test-slave-mrw4 krb5kdc[12958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:40:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:40:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:40:29.708282 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35053
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:35053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:40:30.011941 12971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:30.012212 12971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:30.012327 12971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:30.019531 12971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:40:30.019631 12971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:30.019704 12971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:30.019783 12971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:40:30.019843 12971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:40:30.029585 12971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:35053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:30.030905 12971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:30.032174 12971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:30.043190 12976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:30.043232 12979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:30.043259 12977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:30.043960 12971 server_base.cc:1034] running on GCE node
I20260926 19:40:30.044654 12971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:30.046037 12971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:30.047230 12971 hybrid_clock.cc:648] HybridClock initialized: now 1790451630047178 us; error 43 us; skew 500 ppm
I20260926 19:40:30.047549 12971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:30 dist-test-slave-mrw4 krb5kdc[12958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451630, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:30.050655 12971 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:40:30.052011 12971 webserver.cc:460] Webserver started at http://127.2.164.126:46685/ using document root <none> and password file <none>
I20260926 19:40:30.052691 12971 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:30.052802 12971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:30.053074 12971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:30.055975 12971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "464a4a2c86994aed9c58dd81050538b9"
format_stamp: "Formatted at 2026-09-26 19:40:30 on dist-test-slave-mrw4"
server_key: "e837cdc3545ddb0e45bba65e4ae7c965"
server_key_iv: "ea2c734f697130174674cb5cadc6fc1d"
server_key_version: "encryptionkey@0"
I20260926 19:40:30.056697 12971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "464a4a2c86994aed9c58dd81050538b9"
format_stamp: "Formatted at 2026-09-26 19:40:30 on dist-test-slave-mrw4"
server_key: "e837cdc3545ddb0e45bba65e4ae7c965"
server_key_iv: "ea2c734f697130174674cb5cadc6fc1d"
server_key_version: "encryptionkey@0"
I20260926 19:40:30.061633 12971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 19:40:30.064883 12986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:30.066174 12971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:40:30.066319 12971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "464a4a2c86994aed9c58dd81050538b9"
format_stamp: "Formatted at 2026-09-26 19:40:30 on dist-test-slave-mrw4"
server_key: "e837cdc3545ddb0e45bba65e4ae7c965"
server_key_iv: "ea2c734f697130174674cb5cadc6fc1d"
server_key_version: "encryptionkey@0"
I20260926 19:40:30.066496 12971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:30.084975 12971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:30.088685 12971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:30.088922 12971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:30.104859 12971 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:35053
I20260926 19:40:30.104893 13037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:35053 every 8 connection(s)
I20260926 19:40:30.106153 12971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:40:30.106961 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 12971
I20260926 19:40:30.107177 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:40:30.107712 2705 external_mini_cluster.cc:1428] Setting key c21de7e97e77f1246f918c7460cde34f
I20260926 19:40:30.110093 13038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:40:30 dist-test-slave-mrw4 krb5kdc[12958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:40:30.122090 13038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9: Bootstrap starting.
I20260926 19:40:30.124958 13038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:30.125918 13038 log.cc:824] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9: Log is configured to *not* fsync() on all Append() calls
I20260926 19:40:30.128499 13038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9: No bootstrap required, opened a new log
I20260926 19:40:30.128844 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:30.109530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.2.164.126:35053)
0926 19:40:30.110282 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:30.110294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:30.110353 (+ 59us) server_negotiation.cc:407] Connection header received
0926 19:40:30.111831 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:30.111853 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:30.112200 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:30.112520 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:30.114199 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:30.115490 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:30.117201 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:30.117820 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:30.117948 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:30.122297 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:30.122304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:30.122318 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:30.122367 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:30.125028 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:30.125685 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:30.125691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:30.125697 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:30.125863 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:30.126227 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:30.126230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:30.126232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:30.126780 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:30.127101 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:30.127538 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 19:40:30.127892 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":316,"threads_started":1}
I20260926 19:40:30.136340 13038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a4a2c86994aed9c58dd81050538b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35053 } }
I20260926 19:40:30.136659 13038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:30.136744 13038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464a4a2c86994aed9c58dd81050538b9, State: Initialized, Role: FOLLOWER
I20260926 19:40:30.137305 13038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a4a2c86994aed9c58dd81050538b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35053 } }
I20260926 19:40:30.137517 13038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:40:30.137634 13038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:40:30.137782 13038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:30.139971 13038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a4a2c86994aed9c58dd81050538b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35053 } }
I20260926 19:40:30.140430 13038 leader_election.cc:302] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 464a4a2c86994aed9c58dd81050538b9; no voters:
I20260926 19:40:30.140938 13038 leader_election.cc:288] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:40:30.141041 13043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:40:30.141659 13043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [term 1 LEADER]: Becoming Leader. State: Replica: 464a4a2c86994aed9c58dd81050538b9, State: Running, Role: LEADER
I20260926 19:40:30.142138 13043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a4a2c86994aed9c58dd81050538b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35053 } }
I20260926 19:40:30.142761 13038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:40:30.144647 13044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "464a4a2c86994aed9c58dd81050538b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a4a2c86994aed9c58dd81050538b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35053 } } }
I20260926 19:40:30.144793 13045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464a4a2c86994aed9c58dd81050538b9. Latest consensus state: current_term: 1 leader_uuid: "464a4a2c86994aed9c58dd81050538b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a4a2c86994aed9c58dd81050538b9" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35053 } } }
I20260926 19:40:30.145032 13044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [sys.catalog]: This master's current role is: LEADER
I20260926 19:40:30.145103 13045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9 [sys.catalog]: This master's current role is: LEADER
I20260926 19:40:30.146955 13051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:40:30.151731 13051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:40:30.160460 13051 catalog_manager.cc:1347] Generated new cluster ID: 4f9c3804c3ee4b6791e4bfaec537870e
I20260926 19:40:30.160552 13051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:40:30.168431 13051 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:40:30.169766 13051 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:40:30.175591 13051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9: Generated new TSK 0
I20260926 19:40:30.176313 13051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:40:30.233292 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35053
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:40:30.543049 13064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:30.543335 13064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:30.543478 13064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:30.550625 13064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:30.550719 13064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:30.550860 13064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:40:30.559923 13064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:30.561066 13064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:30.562320 13064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:30.574393 13069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:30.574710 13070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:30.574823 13072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:30.575897 13064 server_base.cc:1034] running on GCE node
I20260926 19:40:30.576453 13064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:30.577369 13064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:30.578589 13064 hybrid_clock.cc:648] HybridClock initialized: now 1790451630578536 us; error 45 us; skew 500 ppm
I20260926 19:40:30.578902 13064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:30 dist-test-slave-mrw4 krb5kdc[12958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451630, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:30.581933 13064 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:40:30.583329 13064 webserver.cc:460] Webserver started at http://127.2.164.65:35817/ using document root <none> and password file <none>
I20260926 19:40:30.583956 13064 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:30.584065 13064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:30.584344 13064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:30.587165 13064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "91c752b6b4bb4522872ffbe097730720"
format_stamp: "Formatted at 2026-09-26 19:40:30 on dist-test-slave-mrw4"
server_key: "a8825a705bcc7341d1f01afac1466325"
server_key_iv: "29055162424fd3d2667ea9921b86d3cd"
server_key_version: "encryptionkey@0"
I20260926 19:40:30.587877 13064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "91c752b6b4bb4522872ffbe097730720"
format_stamp: "Formatted at 2026-09-26 19:40:30 on dist-test-slave-mrw4"
server_key: "a8825a705bcc7341d1f01afac1466325"
server_key_iv: "29055162424fd3d2667ea9921b86d3cd"
server_key_version: "encryptionkey@0"
I20260926 19:40:30.592851 13064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 19:40:30.596230 13079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:30.597510 13064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 19:40:30.597645 13064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "91c752b6b4bb4522872ffbe097730720"
format_stamp: "Formatted at 2026-09-26 19:40:30 on dist-test-slave-mrw4"
server_key: "a8825a705bcc7341d1f01afac1466325"
server_key_iv: "29055162424fd3d2667ea9921b86d3cd"
server_key_version: "encryptionkey@0"
I20260926 19:40:30.597812 13064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:30.618503 13064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:30.621989 13064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:30.622233 13064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:30.623076 13064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:30.624276 13064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:30.624346 13064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:30.624446 13064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:30.624501 13064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:30.653003 13064 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:45355
I20260926 19:40:30.653024 13191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:45355 every 8 connection(s)
I20260926 19:40:30.654223 13064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:40:30.655803 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 13064
I20260926 19:40:30.655987 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:40:30.656725 2705 external_mini_cluster.cc:1428] Setting key 82a8705a71e6596bfbda30d0eb6c490f
Sep 26 19:40:30 dist-test-slave-mrw4 krb5kdc[12958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451630, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:40:30.673419 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:30.656477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:47225 (local address 127.2.164.126:35053)
0926 19:40:30.656915 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:30.656922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:30.657981 (+ 1059us) server_negotiation.cc:407] Connection header received
0926 19:40:30.659873 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:30.659878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:30.659954 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:30.660064 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:30.661576 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:30.662472 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:30.664008 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:30.664540 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:30.664656 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:30.668292 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:30.668297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:30.668300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:30.668331 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:30.670695 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:30.671420 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:30.671423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:30.671426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:30.671534 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:30.671923 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:30.671926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:30.671928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:30.672174 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:30.672276 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:30.673000 (+ 724us) server_negotiation.cc:299] Negotiation successful
0926 19:40:30.673140 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":207,"threads_started":1}
I20260926 19:40:30.674762 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:30.657024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35053 (local address 127.2.164.65:47225)
0926 19:40:30.657821 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:30.657872 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:30.659612 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:30.660302 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:30.660311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:30.660813 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:30.661404 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:30.661423 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:30.662674 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:30.662678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:30.663880 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:30.663892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:30.664750 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:30.664755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:30.664944 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:30.665571 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:30.665611 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:30.668085 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:30.670804 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:30.670812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:30.670826 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:30.671317 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:30.671623 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:30.671627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:30.671631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:30.671829 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:30.672284 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:30.672291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:30.672663 (+ 372us) client_negotiation.cc:769] Sending connection context
0926 19:40:30.672944 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 19:40:30.673432 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":312,"threads_started":1}
I20260926 19:40:30.676141 13192 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35053
I20260926 19:40:30.676412 13192 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:30.677057 13192 heartbeater.cc:511] Master 127.2.164.126:35053 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260926 19:40:30.678978 13003 ts_manager.cc:194] Registered new tserver with Master: 91c752b6b4bb4522872ffbe097730720 (127.2.164.65:45355)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260926 19:40:30.680393 13003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:47225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:40:30.704713 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35053
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:40:31.007807 13198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:31.008064 13198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:31.008163 13198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:31.015175 13198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:31.015272 13198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:31.015389 13198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:40:31.024832 13198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:31.026083 13198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:31.027290 13198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:31.039783 13203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:31.039758 13204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:31.040045 13206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:31.040629 13198 server_base.cc:1034] running on GCE node
I20260926 19:40:31.041280 13198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:31.042204 13198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:31.043421 13198 hybrid_clock.cc:648] HybridClock initialized: now 1790451631043360 us; error 49 us; skew 500 ppm
I20260926 19:40:31.043773 13198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:31 dist-test-slave-mrw4 krb5kdc[12958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:31.051091 13198 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:40:31.052537 13198 webserver.cc:460] Webserver started at http://127.2.164.66:45299/ using document root <none> and password file <none>
I20260926 19:40:31.053269 13198 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:31.053387 13198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:31.053660 13198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:31.056427 13198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "1de2a3d622ca430a802620ba677caa09"
format_stamp: "Formatted at 2026-09-26 19:40:31 on dist-test-slave-mrw4"
server_key: "44539035464d07219c54cbbfcbc8a84a"
server_key_iv: "3f12ee339c87628b494ea4ccc9500644"
server_key_version: "encryptionkey@0"
I20260926 19:40:31.057186 13198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "1de2a3d622ca430a802620ba677caa09"
format_stamp: "Formatted at 2026-09-26 19:40:31 on dist-test-slave-mrw4"
server_key: "44539035464d07219c54cbbfcbc8a84a"
server_key_iv: "3f12ee339c87628b494ea4ccc9500644"
server_key_version: "encryptionkey@0"
I20260926 19:40:31.062083 13198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:40:31.065512 13213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:31.066816 13198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:40:31.066975 13198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "1de2a3d622ca430a802620ba677caa09"
format_stamp: "Formatted at 2026-09-26 19:40:31 on dist-test-slave-mrw4"
server_key: "44539035464d07219c54cbbfcbc8a84a"
server_key_iv: "3f12ee339c87628b494ea4ccc9500644"
server_key_version: "encryptionkey@0"
I20260926 19:40:31.067150 13198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:31.093377 13198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:31.096918 13198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:31.097187 13198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:31.098053 13198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:31.099260 13198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:31.099335 13198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:31.099444 13198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:31.099493 13198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:31.128875 13198 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:37415
I20260926 19:40:31.128913 13325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:37415 every 8 connection(s)
I20260926 19:40:31.130064 13198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:40:31.132752 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 13198
I20260926 19:40:31.132943 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:40:31.133486 2705 external_mini_cluster.cc:1428] Setting key 6e79ba1f6c672d0bb67ee195e1e28260
Sep 26 19:40:31 dist-test-slave-mrw4 krb5kdc[12958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:40:31.148525 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.132361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:42223 (local address 127.2.164.126:35053)
0926 19:40:31.132537 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:31.132542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:31.133857 (+ 1315us) server_negotiation.cc:407] Connection header received
0926 19:40:31.135708 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:31.135713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:31.135807 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:31.135892 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:31.137367 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.138085 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.139538 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.139943 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.140047 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.143290 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:31.143295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:31.143298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:31.143328 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:31.145764 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:31.146485 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:31.146489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:31.146491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:31.146606 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:31.146986 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:31.146991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:31.146994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:31.147244 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:31.147366 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:31.147999 (+ 633us) server_negotiation.cc:299] Negotiation successful
0926 19:40:31.148204 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 19:40:31.149541 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.132937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35053 (local address 127.2.164.66:42223)
0926 19:40:31.133688 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:31.133728 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:31.135471 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:31.136119 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:31.136127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:31.136634 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:31.137207 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.137228 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.138216 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.138221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.139403 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.139418 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.140138 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.140145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.140357 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.140948 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:31.140968 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:31.143109 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:31.145889 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:31.145897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:31.145913 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:31.146380 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:31.146700 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:31.146705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:31.146709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:31.146879 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:31.147356 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:31.147363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:31.147655 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 19:40:31.147871 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 19:40:31.148363 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":290,"threads_started":1}
I20260926 19:40:31.150872 13326 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35053
I20260926 19:40:31.151198 13326 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:31.152071 13326 heartbeater.cc:511] Master 127.2.164.126:35053 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260926 19:40:31.153533 13003 ts_manager.cc:194] Registered new tserver with Master: 1de2a3d622ca430a802620ba677caa09 (127.2.164.66:37415)
I20260926 19:40:31.154294 13003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:42223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:40:31.177085 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35053
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:40:31.482335 13331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:40:31.482583 13331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:40:31.482707 13331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:40:31.489583 13331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:40:31.489681 13331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:40:31.489830 13331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:40:31.499035 13331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:35053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:40:31.500303 13331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:40:31.501698 13331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:40:31.514133 13336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:31.514074 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:40:31.514110 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:40:31.515221 13331 server_base.cc:1034] running on GCE node
I20260926 19:40:31.515759 13331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:40:31.516595 13331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:40:31.517800 13331 hybrid_clock.cc:648] HybridClock initialized: now 1790451631517746 us; error 41 us; skew 500 ppm
I20260926 19:40:31.518082 13331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:40:31 dist-test-slave-mrw4 krb5kdc[12958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:40:31.521004 13331 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:40:31.522397 13331 webserver.cc:460] Webserver started at http://127.2.164.67:45977/ using document root <none> and password file <none>
I20260926 19:40:31.523046 13331 fs_manager.cc:359] Metadata directory not provided
I20260926 19:40:31.523139 13331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:40:31.523375 13331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:40:31.526132 13331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "08fea34090b643098573b6457e9d9491"
format_stamp: "Formatted at 2026-09-26 19:40:31 on dist-test-slave-mrw4"
server_key: "f403e42dec10dd72a94e02c472cd6a09"
server_key_iv: "7d77901e08e8ffe7d1a55f640ef7b4c9"
server_key_version: "encryptionkey@0"
I20260926 19:40:31.526832 13331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "08fea34090b643098573b6457e9d9491"
format_stamp: "Formatted at 2026-09-26 19:40:31 on dist-test-slave-mrw4"
server_key: "f403e42dec10dd72a94e02c472cd6a09"
server_key_iv: "7d77901e08e8ffe7d1a55f640ef7b4c9"
server_key_version: "encryptionkey@0"
I20260926 19:40:31.531908 13331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:40:31.535205 13346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:31.536480 13331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:40:31.536649 13331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "08fea34090b643098573b6457e9d9491"
format_stamp: "Formatted at 2026-09-26 19:40:31 on dist-test-slave-mrw4"
server_key: "f403e42dec10dd72a94e02c472cd6a09"
server_key_iv: "7d77901e08e8ffe7d1a55f640ef7b4c9"
server_key_version: "encryptionkey@0"
I20260926 19:40:31.536814 13331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:40:31.568961 13331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:40:31.572926 13331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:40:31.573308 13331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:40:31.574267 13331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:40:31.575500 13331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:40:31.575569 13331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:31.575659 13331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:40:31.575701 13331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:40:31.604712 13331 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:35413
I20260926 19:40:31.604754 13458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:35413 every 8 connection(s)
I20260926 19:40:31.606051 13331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:40:31.614821 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 13331
I20260926 19:40:31.614995 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:40:31.615476 2705 external_mini_cluster.cc:1428] Setting key de29ce07c63af758836428ee58e74023
Sep 26 19:40:31 dist-test-slave-mrw4 krb5kdc[12958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:40:31.624290 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.608392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:45635 (local address 127.2.164.126:35053)
0926 19:40:31.608609 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:31.608614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:31.609739 (+ 1125us) server_negotiation.cc:407] Connection header received
0926 19:40:31.611422 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:31.611427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:31.611492 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:31.611577 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:31.612918 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.613685 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.615266 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.615846 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.615975 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.619224 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:31.619229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:31.619232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:31.619262 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:31.621592 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:31.622229 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:31.622235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:31.622238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:31.622346 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:31.622737 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:31.622741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:31.622743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:31.622991 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:31.623114 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:31.623758 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 19:40:31.623958 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 19:40:31.625185 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.608910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:35053 (local address 127.2.164.67:45635)
0926 19:40:31.609592 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:31.609629 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:31.611182 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:31.611768 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:31.611776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:31.612255 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:40:31.612781 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.612797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.613847 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.613851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.615123 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.615137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.616069 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.616077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.616338 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.616991 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:40:31.617017 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:40:31.619041 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:40:31.621714 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:31.621723 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:31.621739 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:31.622120 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:31.622442 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:40:31.622447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:40:31.622451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:40:31.622614 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:40:31.623106 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:40:31.623113 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:40:31.623420 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 19:40:31.623658 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 19:40:31.624125 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":300,"threads_started":1}
I20260926 19:40:31.626374 13459 heartbeater.cc:347] Connected to a master server at 127.2.164.126:35053
I20260926 19:40:31.626634 13459 heartbeater.cc:464] Registering TS with master...
I20260926 19:40:31.627317 13459 heartbeater.cc:511] Master 127.2.164.126:35053 requested a full tablet report, sending...
I20260926 19:40:31.628777 13003 ts_manager.cc:194] Registered new tserver with Master: 08fea34090b643098573b6457e9d9491 (127.2.164.67:35413)
I20260926 19:40:31.629742 13003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:45635
I20260926 19:40:31.630522 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:40:31.646292 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.634799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.2.164.126:35053)
0926 19:40:31.634952 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:31.634958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:31.635371 (+ 413us) server_negotiation.cc:407] Connection header received
0926 19:40:31.635474 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:31.635481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:31.635552 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:31.635633 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:31.636972 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.637737 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.638980 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.639400 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.639499 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.641754 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:31.641759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:31.641762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:31.641792 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:31.644157 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:31.644710 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:31.644714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:31.644716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:31.644855 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:31.645150 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:31.645153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:31.645155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:31.645456 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:31.645620 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:31.645914 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 19:40:31.646076 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:40:31.650339 13002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:40:31.657009 13002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:40:31.683198 13192 heartbeater.cc:503] Master 127.2.164.126:35053 was elected leader, sending a full tablet report...
I20260926 19:40:31.692651 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.667894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.2.164.67:35413)
0926 19:40:31.679847 (+ 11953us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:31.679859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:31.682401 (+ 2542us) server_negotiation.cc:407] Connection header received
0926 19:40:31.682514 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:31.682524 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:31.682815 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:31.683014 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:40:31.685611 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.687362 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.690193 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.691556 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.691677 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.691900 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:31.692171 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 19:40:31.692328 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11734,"thread_start_us":11563,"threads_started":1}
I20260926 19:40:31.692672 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.667777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:35413 (local address 127.0.0.1:47310)
0926 19:40:31.681936 (+ 14159us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:31.681981 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:31.682169 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:31.684904 (+ 2735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:31.684913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:31.684945 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:40:31.685392 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.685405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.687538 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.687544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.690023 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.690037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.691729 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.691735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.691925 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.691978 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 19:40:31.692108 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 19:40:31.692288 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13876,"thread_start_us":20815,"threads_started":1}
I20260926 19:40:31.696409 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.668105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.2.164.65:45355)
0926 19:40:31.689237 (+ 21132us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:31.689246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:31.689274 (+ 28us) server_negotiation.cc:407] Connection header received
0926 19:40:31.689361 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:31.689369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:31.689603 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:31.689782 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:40:31.690648 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.692064 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.694566 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.695460 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.695576 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.695792 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:31.695980 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 19:40:31.696126 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20933,"thread_start_us":20955,"threads_started":1}
I20260926 19:40:31.696425 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.668769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:45355 (local address 127.0.0.1:60462)
0926 19:40:31.688846 (+ 20077us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:31.688866 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:31.688963 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:31.690047 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:31.690051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:31.690076 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:40:31.690446 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.690458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.692215 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.692220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.694432 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.694444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.695652 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.695658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.695809 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.695839 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 19:40:31.695921 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 19:40:31.696040 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19959,"mutex_wait_us":39,"thread_start_us":14464,"threads_started":1}
I20260926 19:40:31.696954 13394 tablet_service.cc:1467] Processing CreateTablet for tablet dd1c4795f83441b182836f01f81fd00a (DEFAULT_TABLE table=test-table [id=296457db78d14b77ac2aed0f19362ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:31.698261 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.669032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:37415 (local address 127.0.0.1:57288)
0926 19:40:31.689733 (+ 20701us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:31.689748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:31.689851 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:31.691283 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:31.691287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:31.691310 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:40:31.691660 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.691671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.693490 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.693495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.695487 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.695501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.697477 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.697484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.697659 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.697691 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 19:40:31.697784 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 19:40:31.697900 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20548,"mutex_wait_us":1,"thread_start_us":20410,"threads_started":1}
I20260926 19:40:31.698884 13394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1c4795f83441b182836f01f81fd00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:31.698982 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.669529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.2.164.66:37415)
0926 19:40:31.690858 (+ 21329us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:31.690864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:31.690886 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:40:31.690952 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:31.690957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:31.691118 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:31.691274 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:40:31.691857 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.693343 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.695689 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.697025 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.697158 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.697356 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:31.698519 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0926 19:40:31.698679 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21195,"thread_start_us":21066,"threads_started":1}
I20260926 19:40:31.700057 13127 tablet_service.cc:1467] Processing CreateTablet for tablet dd1c4795f83441b182836f01f81fd00a (DEFAULT_TABLE table=test-table [id=296457db78d14b77ac2aed0f19362ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:31.701215 13127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1c4795f83441b182836f01f81fd00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:31.702472 13261 tablet_service.cc:1467] Processing CreateTablet for tablet dd1c4795f83441b182836f01f81fd00a (DEFAULT_TABLE table=test-table [id=296457db78d14b77ac2aed0f19362ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:40:31.703773 13261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1c4795f83441b182836f01f81fd00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:40:31.712013 13477 tablet_bootstrap.cc:492] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: Bootstrap starting.
I20260926 19:40:31.714964 13477 tablet_bootstrap.cc:654] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:31.715750 13478 tablet_bootstrap.cc:492] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: Bootstrap starting.
I20260926 19:40:31.716193 13477 log.cc:824] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: Log is configured to *not* fsync() on all Append() calls
I20260926 19:40:31.718700 13480 tablet_bootstrap.cc:492] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: Bootstrap starting.
I20260926 19:40:31.719175 13478 tablet_bootstrap.cc:654] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:31.719239 13477 tablet_bootstrap.cc:492] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: No bootstrap required, opened a new log
I20260926 19:40:31.719493 13477 ts_tablet_manager.cc:1397] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 19:40:31.720429 13478 log.cc:824] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: Log is configured to *not* fsync() on all Append() calls
I20260926 19:40:31.721578 13480 tablet_bootstrap.cc:654] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: Neither blocks nor log segments found. Creating new log.
I20260926 19:40:31.722469 13480 log.cc:824] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: Log is configured to *not* fsync() on all Append() calls
I20260926 19:40:31.723214 13478 tablet_bootstrap.cc:492] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: No bootstrap required, opened a new log
I20260926 19:40:31.723467 13478 ts_tablet_manager.cc:1397] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 19:40:31.724674 13480 tablet_bootstrap.cc:492] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: No bootstrap required, opened a new log
I20260926 19:40:31.724884 13480 ts_tablet_manager.cc:1397] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 19:40:31.729249 13477 raft_consensus.cc:348] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08fea34090b643098573b6457e9d9491" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35413 } } peers { permanent_uuid: "91c752b6b4bb4522872ffbe097730720" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45355 } } peers { permanent_uuid: "1de2a3d622ca430a802620ba677caa09" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37415 } }
I20260926 19:40:31.729682 13477 raft_consensus.cc:374] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:31.729775 13477 raft_consensus.cc:729] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c752b6b4bb4522872ffbe097730720, State: Initialized, Role: FOLLOWER
I20260926 19:40:31.730384 13477 consensus_queue.cc:260] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08fea34090b643098573b6457e9d9491" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35413 } } peers { permanent_uuid: "91c752b6b4bb4522872ffbe097730720" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45355 } } peers { permanent_uuid: "1de2a3d622ca430a802620ba677caa09" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37415 } }
I20260926 19:40:31.732084 13477 ts_tablet_manager.cc:1428] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260926 19:40:31.733158 13478 raft_consensus.cc:348] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08fea34090b643098573b6457e9d9491" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35413 } } peers { permanent_uuid: "91c752b6b4bb4522872ffbe097730720" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45355 } } peers { permanent_uuid: "1de2a3d622ca430a802620ba677caa09" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37415 } }
I20260926 19:40:31.733600 13478 raft_consensus.cc:374] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:31.733700 13478 raft_consensus.cc:729] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fea34090b643098573b6457e9d9491, State: Initialized, Role: FOLLOWER
I20260926 19:40:31.733762 13480 raft_consensus.cc:348] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08fea34090b643098573b6457e9d9491" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35413 } } peers { permanent_uuid: "91c752b6b4bb4522872ffbe097730720" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45355 } } peers { permanent_uuid: "1de2a3d622ca430a802620ba677caa09" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37415 } }
I20260926 19:40:31.734139 13480 raft_consensus.cc:374] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:40:31.734222 13480 raft_consensus.cc:729] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de2a3d622ca430a802620ba677caa09, State: Initialized, Role: FOLLOWER
I20260926 19:40:31.734262 13478 consensus_queue.cc:260] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08fea34090b643098573b6457e9d9491" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35413 } } peers { permanent_uuid: "91c752b6b4bb4522872ffbe097730720" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45355 } } peers { permanent_uuid: "1de2a3d622ca430a802620ba677caa09" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37415 } }
I20260926 19:40:31.734719 13480 consensus_queue.cc:260] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08fea34090b643098573b6457e9d9491" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35413 } } peers { permanent_uuid: "91c752b6b4bb4522872ffbe097730720" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45355 } } peers { permanent_uuid: "1de2a3d622ca430a802620ba677caa09" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37415 } }
I20260926 19:40:31.735329 13459 heartbeater.cc:503] Master 127.2.164.126:35053 was elected leader, sending a full tablet report...
I20260926 19:40:31.735998 13478 ts_tablet_manager.cc:1428] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 19:40:31.737068 13326 heartbeater.cc:503] Master 127.2.164.126:35053 was elected leader, sending a full tablet report...
I20260926 19:40:31.737741 13480 ts_tablet_manager.cc:1428] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20260926 19:40:31.857571 13460 tablet.cc:2369] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:40:31.881798 13327 tablet.cc:2369] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:40:31.906315 13193 tablet.cc:2369] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:40:31.951329 13484 raft_consensus.cc:482] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:40:31.951628 13484 raft_consensus.cc:504] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08fea34090b643098573b6457e9d9491" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35413 } } peers { permanent_uuid: "91c752b6b4bb4522872ffbe097730720" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45355 } } peers { permanent_uuid: "1de2a3d622ca430a802620ba677caa09" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37415 } }
I20260926 19:40:31.953291 13484 leader_election.cc:288] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91c752b6b4bb4522872ffbe097730720 (127.2.164.65:45355), 1de2a3d622ca430a802620ba677caa09 (127.2.164.66:37415)
I20260926 19:40:31.959484 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.953748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:43321 (local address 127.2.164.65:45355)
0926 19:40:31.953942 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:31.953948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:31.954001 (+ 53us) server_negotiation.cc:407] Connection header received
0926 19:40:31.954227 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:31.954231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:31.954289 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:31.954362 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:40:31.954764 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.955701 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.957558 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.958689 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.958801 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.958853 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:31.959210 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 19:40:31.959311 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:40:31.959558 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.953729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:45355 (local address 127.2.164.67:43321)
0926 19:40:31.953908 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:31.953935 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:31.954078 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:31.954416 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:31.954421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:31.954456 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:40:31.954618 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.954626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.955823 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.955828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.957412 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.957424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.958825 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.958831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.959019 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.959055 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 19:40:31.959150 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 19:40:31.959279 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 19:40:31.960479 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.953778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:45303 (local address 127.2.164.66:37415)
0926 19:40:31.954115 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:31.954121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:31.954678 (+ 557us) server_negotiation.cc:407] Connection header received
0926 19:40:31.954828 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:31.954832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:31.954911 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:31.954976 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:40:31.955529 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.956463 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.958529 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.959585 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.959720 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.959784 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:31.960121 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 19:40:31.960249 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260926 19:40:31.960549 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.953729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:37415 (local address 127.2.164.67:45303)
0926 19:40:31.954577 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0926 19:40:31.954590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 19:40:31.954706 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:40:31.955016 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:40:31.955021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:40:31.955045 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:40:31.955368 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.955378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.956627 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.956631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.958370 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:40:31.958386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.959843 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.959849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:40:31.959979 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.960001 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:40:31.960082 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 19:40:31.960195 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":115,"thread_start_us":406,"threads_started":1}
I20260926 19:40:31.960516 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1c4795f83441b182836f01f81fd00a" candidate_uuid: "08fea34090b643098573b6457e9d9491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c752b6b4bb4522872ffbe097730720" is_pre_election: true
I20260926 19:40:31.961033 13147 raft_consensus.cc:2454] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08fea34090b643098573b6457e9d9491 in term 0.
I20260926 19:40:31.961525 13281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1c4795f83441b182836f01f81fd00a" candidate_uuid: "08fea34090b643098573b6457e9d9491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1de2a3d622ca430a802620ba677caa09" is_pre_election: true
I20260926 19:40:31.961843 13348 leader_election.cc:302] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08fea34090b643098573b6457e9d9491, 91c752b6b4bb4522872ffbe097730720; no voters:
I20260926 19:40:31.962042 13281 raft_consensus.cc:2454] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08fea34090b643098573b6457e9d9491 in term 0.
I20260926 19:40:31.962287 13484 raft_consensus.cc:2781] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:40:31.962415 13484 raft_consensus.cc:482] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:40:31.962523 13484 raft_consensus.cc:3037] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:31.964655 13484 raft_consensus.cc:504] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08fea34090b643098573b6457e9d9491" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35413 } } peers { permanent_uuid: "91c752b6b4bb4522872ffbe097730720" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45355 } } peers { permanent_uuid: "1de2a3d622ca430a802620ba677caa09" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37415 } }
I20260926 19:40:31.965193 13484 leader_election.cc:288] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [CANDIDATE]: Term 1 election: Requested vote from peers 91c752b6b4bb4522872ffbe097730720 (127.2.164.65:45355), 1de2a3d622ca430a802620ba677caa09 (127.2.164.66:37415)
I20260926 19:40:31.965587 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1c4795f83441b182836f01f81fd00a" candidate_uuid: "08fea34090b643098573b6457e9d9491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c752b6b4bb4522872ffbe097730720"
I20260926 19:40:31.965658 13281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1c4795f83441b182836f01f81fd00a" candidate_uuid: "08fea34090b643098573b6457e9d9491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1de2a3d622ca430a802620ba677caa09"
I20260926 19:40:31.965749 13147 raft_consensus.cc:3037] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:31.965853 13281 raft_consensus.cc:3037] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:40:31.967804 13147 raft_consensus.cc:2454] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08fea34090b643098573b6457e9d9491 in term 1.
I20260926 19:40:31.967919 13281 raft_consensus.cc:2454] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08fea34090b643098573b6457e9d9491 in term 1.
I20260926 19:40:31.968153 13348 leader_election.cc:302] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08fea34090b643098573b6457e9d9491, 91c752b6b4bb4522872ffbe097730720; no voters:
I20260926 19:40:31.968408 13484 raft_consensus.cc:2781] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:40:31.968798 13484 raft_consensus.cc:686] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 1 LEADER]: Becoming Leader. State: Replica: 08fea34090b643098573b6457e9d9491, State: Running, Role: LEADER
I20260926 19:40:31.969261 13484 consensus_queue.cc:237] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08fea34090b643098573b6457e9d9491" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35413 } } peers { permanent_uuid: "91c752b6b4bb4522872ffbe097730720" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45355 } } peers { permanent_uuid: "1de2a3d622ca430a802620ba677caa09" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37415 } }
I20260926 19:40:31.973927 13003 catalog_manager.cc:5495] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08fea34090b643098573b6457e9d9491 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "08fea34090b643098573b6457e9d9491" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08fea34090b643098573b6457e9d9491" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 35413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c752b6b4bb4522872ffbe097730720" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1de2a3d622ca430a802620ba677caa09" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37415 } health_report { overall_health: UNKNOWN } } }
I20260926 19:40:31.991940 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:31.986970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.2.164.67:35413)
0926 19:40:31.987137 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:31.987141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:31.987249 (+ 108us) server_negotiation.cc:407] Connection header received
0926 19:40:31.987423 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:31.987427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:31.987507 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:31.987622 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:40:31.987951 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.988653 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.989987 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:31.990367 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:31.990456 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:31.990904 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:40:31.991409 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:40:31.991518 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:31.991652 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 19:40:31.991727 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:40:31.999614 13281 raft_consensus.cc:1261] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09 [term 1 FOLLOWER]: Refusing update from remote peer 08fea34090b643098573b6457e9d9491: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:40:31.999614 13147 raft_consensus.cc:1261] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720 [term 1 FOLLOWER]: Refusing update from remote peer 08fea34090b643098573b6457e9d9491: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:40:32.000489 13484 consensus_queue.cc:1035] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c752b6b4bb4522872ffbe097730720" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:40:32.000753 13487 consensus_queue.cc:1035] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1de2a3d622ca430a802620ba677caa09" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:40:32.012125 13490 mvcc.cc:205] Tried to move back new op lower bound from 7333689884661571584 to 7333689884549574656. Current Snapshot: MvccSnapshot[applied={T|T < 7333689884661571584}]
I20260926 19:40:32.024747 13003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:40:32.024976 13003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:40:32.028188 13003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 464a4a2c86994aed9c58dd81050538b9: Sending DeleteTablet for 3 replicas of tablet dd1c4795f83441b182836f01f81fd00a
I20260926 19:40:32.029126 13394 tablet_service.cc:1514] Processing DeleteTablet for tablet dd1c4795f83441b182836f01f81fd00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:32 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:47310
I20260926 19:40:32.029246 13127 tablet_service.cc:1514] Processing DeleteTablet for tablet dd1c4795f83441b182836f01f81fd00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:32 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:60462
I20260926 19:40:32.029274 13261 tablet_service.cc:1514] Processing DeleteTablet for tablet dd1c4795f83441b182836f01f81fd00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:40:32 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:57288
I20260926 19:40:32.032516 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 13064
I20260926 19:40:32.042810 13501 tablet_replica.cc:330] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: stopping tablet replica
I20260926 19:40:32.043318 13501 raft_consensus.cc:2229] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:40:32.043999 13501 raft_consensus.cc:2258] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:32.047397 13502 tablet_replica.cc:330] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: stopping tablet replica
I20260926 19:40:32.047755 13502 raft_consensus.cc:2229] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:40:32.048167 13502 raft_consensus.cc:2258] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:40:32.048348 13501 ts_tablet_manager.cc:1905] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:32.050182 13502 ts_tablet_manager.cc:1905] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:32.050431 13503 tablet_replica.cc:330] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: stopping tablet replica
I20260926 19:40:32.050936 13503 raft_consensus.cc:2229] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:40:32.053020 13503 raft_consensus.cc:2258] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 19:40:32 dist-test-slave-mrw4 krb5kdc[12958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:40:32.057319 13503 ts_tablet_manager.cc:1905] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:40:32.057382 13502 ts_tablet_manager.cc:1918] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:40:32.057536 13502 log.cc:1196] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/dd1c4795f83441b182836f01f81fd00a
I20260926 19:40:32.058092 13502 ts_tablet_manager.cc:1939] T dd1c4795f83441b182836f01f81fd00a P 1de2a3d622ca430a802620ba677caa09: Deleting consensus metadata
I20260926 19:40:32.058384 13501 ts_tablet_manager.cc:1918] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:40:32.058581 13501 log.cc:1196] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/dd1c4795f83441b182836f01f81fd00a
I20260926 19:40:32.059170 13501 ts_tablet_manager.cc:1939] T dd1c4795f83441b182836f01f81fd00a P 08fea34090b643098573b6457e9d9491: Deleting consensus metadata
I20260926 19:40:32.060009 12989 catalog_manager.cc:4841] TS 1de2a3d622ca430a802620ba677caa09 (127.2.164.66:37415): tablet dd1c4795f83441b182836f01f81fd00a (table test-table [id=296457db78d14b77ac2aed0f19362ddf]) successfully deleted
I20260926 19:40:32.060827 12990 catalog_manager.cc:4841] TS 08fea34090b643098573b6457e9d9491 (127.2.164.67:35413): tablet dd1c4795f83441b182836f01f81fd00a (table test-table [id=296457db78d14b77ac2aed0f19362ddf]) successfully deleted
I20260926 19:40:32.063294 13503 ts_tablet_manager.cc:1918] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:40:32.063517 13503 log.cc:1196] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/dd1c4795f83441b182836f01f81fd00a
I20260926 19:40:32.064036 13503 ts_tablet_manager.cc:1939] T dd1c4795f83441b182836f01f81fd00a P 91c752b6b4bb4522872ffbe097730720: Deleting consensus metadata
I20260926 19:40:32.065774 12988 catalog_manager.cc:4841] TS 91c752b6b4bb4522872ffbe097730720 (127.2.164.65:45355): tablet dd1c4795f83441b182836f01f81fd00a (table test-table [id=296457db78d14b77ac2aed0f19362ddf]) successfully deleted
I20260926 19:40:32.068626 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:32.033515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.2.164.65:45355)
0926 19:40:32.033710 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:32.033717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:32.039728 (+ 6011us) server_negotiation.cc:407] Connection header received
0926 19:40:32.039927 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:32.039934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:32.040027 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:32.040151 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:32.045772 (+ 5621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:32.047051 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:32.051153 (+ 4102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:32.051782 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:32.051878 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:32.062402 (+ 10524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:32.062415 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:32.062433 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:32.062478 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:32.065982 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:32.066809 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:32.066816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:32.066818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:32.066937 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:32.067341 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:32.067344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:32.067347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:32.067669 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:32.067883 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:32.068218 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 19:40:32.068366 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:40:32.069473 13187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:32.207885 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 13064
I20260926 19:40:32.222293 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 13198
Sep 26 19:40:32 dist-test-slave-mrw4 krb5kdc[12958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:40:32.238103 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:32.223100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.2.164.66:37415)
0926 19:40:32.223293 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:32.223299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:32.223320 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:40:32.223392 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:32.223398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:32.223477 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:32.223586 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:32.224711 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:32.225870 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:32.227013 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:32.227611 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:32.227729 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:32.231771 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:32.231791 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:32.231814 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:32.231870 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:32.235334 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:32.235935 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:32.235946 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:32.235949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:32.236090 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:32.236471 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:32.236476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:32.236479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:32.236891 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:32.237154 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:32.237596 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 19:40:32.237798 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 19:40:32.238797 13321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:32.377883 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 13198
I20260926 19:40:32.397599 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 13331
Sep 26 19:40:32 dist-test-slave-mrw4 krb5kdc[12958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:40:32.414968 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:40:32.398346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.2.164.67:35413)
0926 19:40:32.398516 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:40:32.398522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:40:32.398740 (+ 218us) server_negotiation.cc:407] Connection header received
0926 19:40:32.398969 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:40:32.398973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:40:32.399088 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:40:32.399180 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:40:32.400875 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:32.401684 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:32.403430 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:40:32.403862 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:40:32.403957 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:40:32.409117 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:40:32.409131 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:40:32.409148 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:40:32.409213 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:40:32.412032 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:32.412818 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:32.412825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:32.412827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:32.412968 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:40:32.413565 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:40:32.413568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:40:32.413570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:40:32.413929 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:40:32.414152 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:40:32.414485 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 19:40:32.414627 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 19:40:32.415656 13454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:32.534307 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 13331
I20260926 19:40:32.549842 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 12971
I20260926 19:40:32.550907 13033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:40:32.655071 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 12971
2026-09-26T19:40:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4154 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:40:32 dist-test-slave-mrw4 krb5kdc[13511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:40:32 dist-test-slave-mrw4 krb5kdc[13511](info): set up 2 sockets
Sep 26 19:40:32 dist-test-slave-mrw4 krb5kdc[13511](info): commencing operation
krb5kdc: starting...
W20260926 19:40:33.661811 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:40:33 dist-test-slave-mrw4 krb5kdc[13511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:40:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:40:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/rangeradmin_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/rangeradmin_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/rangerlookup_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/rangerlookup_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
I20260926 19:40:33.950387 2705 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDi4fXZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/postgres -l logfile start
2026-09-26 19:40:36.362 UTC [13541] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 19:40:36.362 UTC [13541] LOG: listening on IPv4 address "127.2.164.84", port 60411
2026-09-26 19:40:36.365 UTC [13541] LOG: listening on Unix socket "/tmp/.s.PGSQL.60411"
2026-09-26 19:40:36.387 UTC [13544] LOG: database system was shut down at 2026-09-26 19:40:34 UTC
2026-09-26 19:40:36.390 UTC [13541] LOG: database system is ready to accept connections
I20260926 19:40:37.273543 2705 mini_postgres.cc:96] Postgres bound to 60411
2026-09-26 19:40:37.289 UTC [13552] FATAL: database "slave" does not exist
127.2.164.84:60411 - accepting connections
I20260926 19:40:37.290700 2705 mini_ranger.cc:161] Starting Ranger...
I20260926 19:40:37.315979 2705 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 19:40:37.816552 2705 mini_ranger.cc:87] Created ranger Postgres database
I20260926 19:40:37.816665 2705 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-admin
2026-09-26 19:40:38,092 [I] DB FLAVOR :POSTGRES
2026-09-26 19:40:38,092 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 19:40:38,092 [I] Checking connection..
2026-09-26 19:40:38,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 19:40:38,316 [I] Checking connection passed.
2026-09-26 19:40:38,316 [I] --------- Verifying version history table ---------
2026-09-26 19:40:38,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:40:38,568 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 19:40:38,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:40:38,814 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 19:40:38,814 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 19:40:38,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 19:40:39.061 UTC [13616] WARNING: there is no transaction in progress
2026-09-26 19:40:39,067 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 19:40:39,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:40:39,322 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 19:40:39,322 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 19:40:39,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 19:40:39,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 19:40:39,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mrw4.c.gcp-upstream.internal','N') ;"
2026-09-26 19:40:40,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 19:40:40,255 [I] Table x_portal_user does not exist in database ranger
2026-09-26 19:40:40,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 19:40:40,485 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 19:40:40,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 19:40:40,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 19:40:40,970 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 19:40:40,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 19:40:41.923 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:41.940 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:41.958 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:41.974 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:41.991 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42.010 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42.066 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42.073 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42.089 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42.100 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42.110 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42.123 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42.133 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42.140 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42.148 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42.155 UTC [13724] WARNING: there is no transaction in progress
2026-09-26 19:40:42,837 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 19:40:42,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 19:40:43,079 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 19:40:43,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 19:40:43,331 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 19:40:43,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 19:40:43,587 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 19:40:43,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 19:40:43,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 19:40:44,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mrw4.c.gcp-upstream.internal';"
2026-09-26 19:40:44,279 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 19:40:44,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 19:40:44,523 [I] DB_PATCHES have already been applied
I20260926 19:40:44.528962 2705 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDi4fXZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 19:40:44.529047 2705 mini_ranger.cc:193] Using host: 127.2.164.84
I20260926 19:40:44.542403 2705 mini_ranger.cc:227] Ranger admin URL: http://127.2.164.84:45749
Sep 26, 2026 7:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:40:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:40:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:40:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 19:40:45.544915 2705 mini_ranger.cc:160] Time spent starting Ranger: real 8.254s user 0.003s sys 0.034s
Sep 26, 2026 7:41:03 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:41:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 7:41:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:41:03 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:41:04 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 19:41:05.359421 2705 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/rangerkms_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/rangerkms_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.164.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.164.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 19:41:05 dist-test-slave-mrw4 krb5kdc[13511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451665, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 19:41:05.503875 2705 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 19:41:05.523169 2705 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 19:41:06.339265 2705 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 19:41:06.339423 2705 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-kms
I20260926 19:41:06.339555 2705 mini_ranger_kms.cc:227] Using postgres at 127.2.164.84:60411
2026-09-26 19:41:06,725 [I] DB FLAVOR :POSTGRES
2026-09-26 19:41:06,725 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 19:41:06,726 [I] Checking connection
2026-09-26 19:41:06,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 19:41:06,957 [I] connection success
2026-09-26 19:41:06,958 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 19:41:06,958 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 19:41:06,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 19:41:07,182 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 19:41:07,183 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 19:41:07,183 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 19:41:07,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:60411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 19:41:07,433 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 19:41:07.905934 2705 mini_ranger_kms.cc:310] Created kms service
I20260926 19:41:08.096446 2705 mini_ranger_kms.cc:326] Created kudu user
I20260926 19:41:08.138646 2705 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 19:41:08 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451665, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:41:08.522687 2705 mini_ranger_kms.cc:384] Added ranger policy
I20260926 19:41:08.522972 2705 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 19:41:08.523027 2705 mini_ranger_kms.cc:240] Using host: 127.2.164.84
I20260926 19:41:08.538784 2705 mini_ranger_kms.cc:276] Ranger KMS PID: 13904
I20260926 19:41:08.538968 2705 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.164.84:40611
Sep 26, 2026 7:41:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:41:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:41:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:41:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:41:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:41:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 19:41:10.610806 2705 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.107s user 0.106s sys 0.294s
I20260926 19:41:10.611017 2705 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 19:41:10.611202 2705 mini_ranger_kms.cc:401] 127.2.164.84:40611/kms/v1/keys
Sep 26 19:41:10 dist-test-slave-mrw4 krb5kdc[13511](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790451670, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 19:41:11 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451670, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
Sep 26 19:41:11 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451670, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
Sep 26 19:41:13 dist-test-slave-mrw4 krb5kdc[13511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:41:13.808374 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:46059
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:46059
--ranger_config_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:40611/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:41:14.128561 13955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:41:14.128845 13955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:41:14.128965 13955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:41:14.135993 13955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:41:14.136109 13955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:41:14.136185 13955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:41:14.136253 13955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:41:14.136315 13955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:41:14.146219 13955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.164.84:40611/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:46059
--ranger_config_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:41:14.147790 13955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:41:14.149279 13955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:41:14.163326 13963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:41:14.163857 13961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:41:14.163810 13960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:41:14.164405 13955 server_base.cc:1034] running on GCE node
I20260926 19:41:14.165088 13955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:41:14.166385 13955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:41:14.167589 13955 hybrid_clock.cc:648] HybridClock initialized: now 1790451674167537 us; error 44 us; skew 500 ppm
I20260926 19:41:14.167927 13955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:41:14 dist-test-slave-mrw4 krb5kdc[13511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:41:14.173942 13955 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:41:14.175379 13955 webserver.cc:460] Webserver started at http://127.2.164.126:34175/ using document root <none> and password file <none>
I20260926 19:41:14.175977 13955 fs_manager.cc:359] Metadata directory not provided
I20260926 19:41:14.176072 13955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:41:14.176345 13955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:41:14 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451674, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:41:14.224649 13955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "6463065f452c4f518f0940f1ddf6b106"
format_stamp: "Formatted at 2026-09-26 19:41:14 on dist-test-slave-mrw4"
server_key: "9b0770fdc9266b202e078bfcb807d61b"
server_key_iv: "cdb4274cfc9f9b75d31367e1f522faf5"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:14.225883 13955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "6463065f452c4f518f0940f1ddf6b106"
format_stamp: "Formatted at 2026-09-26 19:41:14 on dist-test-slave-mrw4"
server_key: "9b0770fdc9266b202e078bfcb807d61b"
server_key_iv: "cdb4274cfc9f9b75d31367e1f522faf5"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:14.230898 13955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:41:14.250012 13970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:41:14.251408 13955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:41:14.251555 13955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "6463065f452c4f518f0940f1ddf6b106"
format_stamp: "Formatted at 2026-09-26 19:41:14 on dist-test-slave-mrw4"
server_key: "9b0770fdc9266b202e078bfcb807d61b"
server_key_iv: "cdb4274cfc9f9b75d31367e1f522faf5"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:14.251746 13955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:41:14.272258 13955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:41:14.276394 13955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:41:14.276656 13955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:41:14.295089 13955 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:46059
I20260926 19:41:14.295092 14021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:46059 every 8 connection(s)
I20260926 19:41:14.296397 13955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:41:14.301612 14022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:41:14.304011 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 13955
I20260926 19:41:14.304208 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:41:14.304821 2705 external_mini_cluster.cc:1428] Setting key b12d5ad7e30c410a042da1d6922dfc31
I20260926 19:41:14.315936 14022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106: Bootstrap starting.
I20260926 19:41:14.319808 14022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106: Neither blocks nor log segments found. Creating new log.
I20260926 19:41:14.321182 14022 log.cc:824] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106: Log is configured to *not* fsync() on all Append() calls
I20260926 19:41:14.325317 14022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106: No bootstrap required, opened a new log
Sep 26 19:41:14 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:41:14.336354 14022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6463065f452c4f518f0940f1ddf6b106" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46059 } }
I20260926 19:41:14.336809 14022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:41:14.336917 14022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6463065f452c4f518f0940f1ddf6b106, State: Initialized, Role: FOLLOWER
I20260926 19:41:14.337723 14022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6463065f452c4f518f0940f1ddf6b106" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46059 } }
I20260926 19:41:14.338021 14022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:41:14.338225 14022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:41:14.338430 14022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:41:14.340313 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:14.306862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.2.164.126:46059)
0926 19:41:14.322329 (+ 15467us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:14.322341 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:14.322384 (+ 43us) server_negotiation.cc:407] Connection header received
0926 19:41:14.324265 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:14.324288 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:14.324652 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:14.324985 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:41:14.326234 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:14.327500 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:14.328699 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:14.329428 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:14.329584 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:14.333018 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:41:14.333025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:41:14.333039 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:41:14.333077 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:41:14.336202 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:14.336855 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:14.336864 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:14.336872 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:14.337109 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:14.337542 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:14.337547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:14.337551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:14.338134 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:41:14.338409 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:14.338903 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 19:41:14.339333 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15265,"thread_start_us":14347,"threads_started":1}
I20260926 19:41:14.341802 14022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6463065f452c4f518f0940f1ddf6b106" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46059 } }
I20260926 19:41:14.342406 14022 leader_election.cc:302] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6463065f452c4f518f0940f1ddf6b106; no voters:
I20260926 19:41:14.343197 14022 leader_election.cc:288] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:41:14.343282 14027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:41:14.344393 14027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [term 1 LEADER]: Becoming Leader. State: Replica: 6463065f452c4f518f0940f1ddf6b106, State: Running, Role: LEADER
I20260926 19:41:14.344899 14027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6463065f452c4f518f0940f1ddf6b106" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46059 } }
I20260926 19:41:14.345397 14022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:41:14.347402 14029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6463065f452c4f518f0940f1ddf6b106. Latest consensus state: current_term: 1 leader_uuid: "6463065f452c4f518f0940f1ddf6b106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6463065f452c4f518f0940f1ddf6b106" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46059 } } }
I20260926 19:41:14.347785 14029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [sys.catalog]: This master's current role is: LEADER
I20260926 19:41:14.348645 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6463065f452c4f518f0940f1ddf6b106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6463065f452c4f518f0940f1ddf6b106" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46059 } } }
I20260926 19:41:14.348908 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106 [sys.catalog]: This master's current role is: LEADER
I20260926 19:41:14.357780 14035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:41:14.363344 14035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:41:14.363994 14022 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 19:41:14.385311 14035 catalog_manager.cc:1347] Generated new cluster ID: 6921b284140e4ee1a6fd9bb4b6f02c21
I20260926 19:41:14.385732 14035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:41:14.407881 14035 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:41:14.410010 14035 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:41:14.427655 14035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106: Generated new TSK 0
I20260926 19:41:14.428848 14035 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:41:15.298894 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:40611/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46059
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:41:15 dist-test-slave-mrw4 krb5kdc[13511](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790451675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 19:41:15.665949 14067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:41:15.666222 14067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:41:15.666386 14067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:41:15.674041 14067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:41:15.674155 14067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:41:15.674309 14067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:41:15.684268 14067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.164.84:40611/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:41:15.685732 14067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:41:15.686952 14067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:41:15.704133 14077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:41:15.706614 14078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:41:15.724575 14067 server_base.cc:1034] running on GCE node
W20260926 19:41:15.725891 14080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:41:15.726854 14067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:41:15.728235 14067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:41:15.729493 14067 hybrid_clock.cc:648] HybridClock initialized: now 1790451675729421 us; error 60 us; skew 500 ppm
I20260926 19:41:15.729826 14067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:41:15 dist-test-slave-mrw4 krb5kdc[13511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:41:15.732894 14067 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:41:15.740521 14067 webserver.cc:460] Webserver started at http://127.2.164.65:38541/ using document root <none> and password file <none>
I20260926 19:41:15.741252 14067 fs_manager.cc:359] Metadata directory not provided
I20260926 19:41:15.741371 14067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:41:15.741679 14067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:41:15 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:41:15.761427 14067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3"
format_stamp: "Formatted at 2026-09-26 19:41:15 on dist-test-slave-mrw4"
server_key: "8f6e44a510a5af5db01b4283d2af2d2e"
server_key_iv: "8a001ed70f2dc5c52c83ba8b4ba8b7de"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:15.762813 14067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3"
format_stamp: "Formatted at 2026-09-26 19:41:15 on dist-test-slave-mrw4"
server_key: "8f6e44a510a5af5db01b4283d2af2d2e"
server_key_iv: "8a001ed70f2dc5c52c83ba8b4ba8b7de"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:15.782155 14067 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.013s sys 0.004s
I20260926 19:41:15.815755 14087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:41:15.822705 14067 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.006s sys 0.005s
I20260926 19:41:15.822913 14067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3"
format_stamp: "Formatted at 2026-09-26 19:41:15 on dist-test-slave-mrw4"
server_key: "8f6e44a510a5af5db01b4283d2af2d2e"
server_key_iv: "8a001ed70f2dc5c52c83ba8b4ba8b7de"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:15.823297 14067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:41:15.849805 14067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:41:15.854789 14067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:41:15.855125 14067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:41:15.865477 14067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:41:15.876869 14067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:41:15.877012 14067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:41:15.877207 14067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:41:15.877316 14067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:41:16 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:41:16.255887 14067 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:34743
I20260926 19:41:16.255900 14199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:34743 every 8 connection(s)
I20260926 19:41:16.257294 14067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:41:16.260650 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 14067
I20260926 19:41:16.260870 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:41:16.262370 2705 external_mini_cluster.cc:1428] Setting key a5446e8f3a8f85779a3168a9f8850704
Sep 26 19:41:16 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:41:16.294452 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:16.273023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:41957 (local address 127.2.164.126:46059)
0926 19:41:16.278957 (+ 5934us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:16.278965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:16.278986 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:41:16.279053 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:16.279059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:16.279150 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:16.279257 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:41:16.282414 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:16.283415 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:16.285395 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:16.285785 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:16.285892 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:16.288927 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:41:16.288932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:41:16.288935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:41:16.288966 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:41:16.291513 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:16.292243 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:16.292249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:16.292251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:16.292367 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:16.292781 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:16.292786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:16.292788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:16.293039 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:41:16.293189 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:16.293868 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 19:41:16.294089 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5780,"thread_start_us":5648,"threads_started":1}
I20260926 19:41:16.295465 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:16.272334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:46059 (local address 127.2.164.65:41957)
0926 19:41:16.275325 (+ 2991us) negotiation.cc:107] Waiting for socket to connect
0926 19:41:16.275395 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 19:41:16.278177 (+ 2782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:41:16.280073 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:41:16.280081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:41:16.280521 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:41:16.282197 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:16.282235 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:16.283558 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:16.283564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:16.285244 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:16.285269 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:16.285992 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:16.285999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:16.286212 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:16.286823 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:41:16.286845 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:41:16.288711 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:41:16.291643 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:41:16.291653 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:41:16.291670 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:41:16.292131 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:41:16.292465 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:41:16.292470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:41:16.292474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:41:16.292670 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:41:16.293192 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:41:16.293210 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:41:16.293527 (+ 317us) client_negotiation.cc:769] Sending connection context
0926 19:41:16.293759 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 19:41:16.294277 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2635,"thread_start_us":2371,"threads_started":1}
I20260926 19:41:16.296978 14200 heartbeater.cc:347] Connected to a master server at 127.2.164.126:46059
I20260926 19:41:16.297289 14200 heartbeater.cc:464] Registering TS with master...
I20260926 19:41:16.297951 14200 heartbeater.cc:511] Master 127.2.164.126:46059 requested a full tablet report, sending...
I20260926 19:41:16.299945 13987 ts_manager.cc:194] Registered new tserver with Master: dcdcc1a66b2e494cba2f83d2eb6067d3 (127.2.164.65:34743)
I20260926 19:41:16.301544 13987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:41957
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:41:16.343856 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:40611/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46059
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:41:16 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451675, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
W20260926 19:41:16.822623 14206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:41:16.823143 14206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:41:16.823297 14206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:41:16.830641 14206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:41:16.830755 14206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:41:16.830904 14206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:41:16.841138 14206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.164.84:40611/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:41:16.842463 14206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:41:16.843713 14206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:41:16.858664 14221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:41:16.858880 14223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:41:16.859393 14220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:41:16.860255 14206 server_base.cc:1034] running on GCE node
I20260926 19:41:16.860798 14206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:41:16.861763 14206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:41:16.862970 14206 hybrid_clock.cc:648] HybridClock initialized: now 1790451676862915 us; error 49 us; skew 500 ppm
I20260926 19:41:16.863247 14206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:41:16 dist-test-slave-mrw4 krb5kdc[13511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451676, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:41:16.866319 14206 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:41:16.867758 14206 webserver.cc:460] Webserver started at http://127.2.164.66:34465/ using document root <none> and password file <none>
I20260926 19:41:16.868372 14206 fs_manager.cc:359] Metadata directory not provided
I20260926 19:41:16.868482 14206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:41:16.868757 14206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:41:16 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451676, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:41:16.886901 14206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "873e8178a72c4185bc5b01f7422016e5"
format_stamp: "Formatted at 2026-09-26 19:41:16 on dist-test-slave-mrw4"
server_key: "ca637e10df79144861cc3782fab98b5a"
server_key_iv: "6b64a0367866f5794d70e939a03a7f4e"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:16.888201 14206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "873e8178a72c4185bc5b01f7422016e5"
format_stamp: "Formatted at 2026-09-26 19:41:16 on dist-test-slave-mrw4"
server_key: "ca637e10df79144861cc3782fab98b5a"
server_key_iv: "6b64a0367866f5794d70e939a03a7f4e"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:16.896898 14206 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260926 19:41:16.922117 14230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:41:16.923623 14206 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.008s sys 0.003s
I20260926 19:41:16.923791 14206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "873e8178a72c4185bc5b01f7422016e5"
format_stamp: "Formatted at 2026-09-26 19:41:16 on dist-test-slave-mrw4"
server_key: "ca637e10df79144861cc3782fab98b5a"
server_key_iv: "6b64a0367866f5794d70e939a03a7f4e"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:16.924012 14206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:41:16.960703 14206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:41:16.965098 14206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:41:16.965392 14206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:41:16.966485 14206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:41:16.967895 14206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:41:16.967968 14206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:41:16.968107 14206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:41:16.968158 14206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:41:17.010557 14206 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:35227
I20260926 19:41:17.010572 14342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:35227 every 8 connection(s)
I20260926 19:41:17.011909 14206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:41:17.017486 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 14206
I20260926 19:41:17.017657 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:41:17.018261 2705 external_mini_cluster.cc:1428] Setting key e049543af5533e624be61da8d093a170
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Sep 26 19:41:17 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451676, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:41:17.065819 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.016859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:34167 (local address 127.2.164.126:46059)
0926 19:41:17.041481 (+ 24622us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:17.041488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:17.046166 (+ 4678us) server_negotiation.cc:407] Connection header received
0926 19:41:17.046677 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:17.046684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:17.046792 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:17.046882 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:41:17.048894 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.049671 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.052574 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.053144 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.053283 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.059262 (+ 5979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:41:17.059266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:41:17.059269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:41:17.059301 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:41:17.062404 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:17.063255 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:17.063261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:17.063264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:17.063392 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:17.063879 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:17.063884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:17.063887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:17.064198 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:41:17.064313 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:17.065268 (+ 955us) server_negotiation.cc:299] Negotiation successful
0926 19:41:17.065473 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24476,"thread_start_us":29321,"threads_started":1}
I20260926 19:41:17.068390 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.016130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:46059 (local address 127.2.164.66:34167)
0926 19:41:17.042103 (+ 25973us) negotiation.cc:107] Waiting for socket to connect
0926 19:41:17.042164 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 19:41:17.045136 (+ 2972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:41:17.047531 (+ 2395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:41:17.047543 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:41:17.048105 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:41:17.048721 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.048749 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.049826 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.049830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.052394 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.052418 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.053565 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.053571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.053864 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.054598 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:41:17.054626 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:41:17.059052 (+ 4426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:41:17.062564 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:41:17.062573 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:41:17.062592 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:41:17.063118 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:41:17.063517 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:41:17.063521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:41:17.063524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:41:17.063762 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:41:17.064412 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:41:17.064419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:41:17.064827 (+ 408us) client_negotiation.cc:769] Sending connection context
0926 19:41:17.066350 (+ 1523us) client_negotiation.cc:240] Negotiation successful
0926 19:41:17.066962 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25618,"thread_start_us":28935,"threads_started":1}
I20260926 19:41:17.070358 14343 heartbeater.cc:347] Connected to a master server at 127.2.164.126:46059
I20260926 19:41:17.070673 14343 heartbeater.cc:464] Registering TS with master...
I20260926 19:41:17.071496 14343 heartbeater.cc:511] Master 127.2.164.126:46059 requested a full tablet report, sending...
I20260926 19:41:17.073179 13987 ts_manager.cc:194] Registered new tserver with Master: 873e8178a72c4185bc5b01f7422016e5 (127.2.164.66:35227)
I20260926 19:41:17.074221 13987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:34167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:41:17.082670 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:40611/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46059
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:41:17.305094 14200 heartbeater.cc:503] Master 127.2.164.126:46059 was elected leader, sending a full tablet report...
W20260926 19:41:17.428769 14349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:41:17.429059 14349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:41:17.429193 14349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:41:17.436008 14349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:41:17.436097 14349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:41:17.436233 14349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:41:17.445642 14349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.164.84:40611/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:41:17.446801 14349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:41:17.447974 14349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:41:17.466329 14354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:41:17.466255 14355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:41:17.466298 14357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:41:17.467451 14349 server_base.cc:1034] running on GCE node
I20260926 19:41:17.467974 14349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:41:17.468865 14349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:41:17.470045 14349 hybrid_clock.cc:648] HybridClock initialized: now 1790451677470003 us; error 36 us; skew 500 ppm
I20260926 19:41:17.470327 14349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:41:17 dist-test-slave-mrw4 krb5kdc[13511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451677, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:41:17.473017 14349 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:41:17.474400 14349 webserver.cc:460] Webserver started at http://127.2.164.67:46807/ using document root <none> and password file <none>
I20260926 19:41:17.475000 14349 fs_manager.cc:359] Metadata directory not provided
I20260926 19:41:17.475102 14349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:41:17.475373 14349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:41:17 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451677, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:41:17.488093 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "74ccfd53f595477596c73681c0a7e1f3"
format_stamp: "Formatted at 2026-09-26 19:41:17 on dist-test-slave-mrw4"
server_key: "d0d70c50df37c2c4915f1575eb1611a7"
server_key_iv: "5a7c69d9accca15ceee3fbe8a640f7ac"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:17.488811 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "74ccfd53f595477596c73681c0a7e1f3"
format_stamp: "Formatted at 2026-09-26 19:41:17 on dist-test-slave-mrw4"
server_key: "d0d70c50df37c2c4915f1575eb1611a7"
server_key_iv: "5a7c69d9accca15ceee3fbe8a640f7ac"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:17.493593 14349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260926 19:41:17.507001 14364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:41:17.508415 14349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:41:17.508554 14349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "74ccfd53f595477596c73681c0a7e1f3"
format_stamp: "Formatted at 2026-09-26 19:41:17 on dist-test-slave-mrw4"
server_key: "d0d70c50df37c2c4915f1575eb1611a7"
server_key_iv: "5a7c69d9accca15ceee3fbe8a640f7ac"
server_key_version: "kuduclusterkey@0"
I20260926 19:41:17.508754 14349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:41:17.521874 14349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:41:17.525669 14349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:41:17.525933 14349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:41:17.526847 14349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:41:17.528218 14349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:41:17.528288 14349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:41:17.528395 14349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:41:17.528450 14349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:41:17.560168 14349 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:39929
I20260926 19:41:17.560213 14476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:39929 every 8 connection(s)
I20260926 19:41:17.561434 14349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:41:17.570611 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 14349
I20260926 19:41:17.570762 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:41:17.571183 2705 external_mini_cluster.cc:1428] Setting key fafd267af51de8eebb753f5fc13c3b8d
Sep 26 19:41:17 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451677, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:41:17.579344 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.563694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:34353 (local address 127.2.164.126:46059)
0926 19:41:17.563844 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:17.563848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:17.564862 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 19:41:17.566549 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:17.566554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:17.566648 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:17.566730 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:41:17.568057 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.568739 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.570252 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.570649 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.570754 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.574342 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:41:17.574346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:41:17.574350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:41:17.574378 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:41:17.576796 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:17.577463 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:17.577467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:17.577469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:17.577593 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:17.577950 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:17.577954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:17.577956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:17.578237 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:41:17.578346 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:17.578892 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 19:41:17.579065 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 19:41:17.580098 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.564093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:46059 (local address 127.2.164.67:34353)
0926 19:41:17.564718 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0926 19:41:17.564753 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 19:41:17.566294 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:41:17.566948 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:41:17.566956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:41:17.567353 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:41:17.567902 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.567918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.568883 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.568886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.570108 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.570125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.570839 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.570846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.571052 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.571692 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:41:17.571719 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:41:17.574155 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:41:17.576909 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:41:17.576915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:41:17.576927 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:41:17.577349 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:41:17.577673 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:41:17.577676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:41:17.577678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:41:17.577827 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:41:17.578326 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:41:17.578332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:41:17.578595 (+ 263us) client_negotiation.cc:769] Sending connection context
0926 19:41:17.578800 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 19:41:17.579125 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":233,"threads_started":1}
I20260926 19:41:17.581306 14477 heartbeater.cc:347] Connected to a master server at 127.2.164.126:46059
I20260926 19:41:17.581529 14477 heartbeater.cc:464] Registering TS with master...
I20260926 19:41:17.582165 14477 heartbeater.cc:511] Master 127.2.164.126:46059 requested a full tablet report, sending...
I20260926 19:41:17.583413 13987 ts_manager.cc:194] Registered new tserver with Master: 74ccfd53f595477596c73681c0a7e1f3 (127.2.164.67:39929)
I20260926 19:41:17.584188 13987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:34353
I20260926 19:41:17.586038 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:41:17.602108 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.590148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.2.164.126:46059)
0926 19:41:17.590287 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:17.590291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:17.590714 (+ 423us) server_negotiation.cc:407] Connection header received
0926 19:41:17.590887 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:17.590893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:17.590959 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:17.591039 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:41:17.592395 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.593071 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.594789 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.595198 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.595311 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.597555 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:41:17.597559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:41:17.597563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:41:17.597593 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:41:17.599765 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:17.600375 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:17.600381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:17.600385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:17.600497 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:17.600828 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:17.600832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:17.600836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:17.601081 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:41:17.601352 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:17.601602 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 19:41:17.601792 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 19:41:17.605623 13987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:41:17.612573 13987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:41:17.633910 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.624387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.2.164.66:35227)
0926 19:41:17.625542 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:17.625551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:17.625576 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:41:17.625673 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:17.625680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:17.625933 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:17.626806 (+ 873us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:41:17.627022 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.629029 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.631564 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.632823 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.632958 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.633151 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:17.633458 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 19:41:17.633625 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":886,"threads_started":1}
I20260926 19:41:17.633955 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.624180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:35227 (local address 127.0.0.1:36564)
0926 19:41:17.624931 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 19:41:17.624971 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:41:17.625130 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:41:17.626141 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:41:17.626148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:41:17.626199 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:41:17.626592 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.626608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.629184 (+ 2576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.629190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.631394 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.631407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.633016 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.633021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.633222 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.633272 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 19:41:17.633403 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 19:41:17.633593 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":375,"threads_started":1}
I20260926 19:41:17.634433 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.625454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:34743 (local address 127.0.0.1:50594)
0926 19:41:17.625942 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 19:41:17.625962 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:41:17.626064 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:41:17.627090 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:41:17.627095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:41:17.627120 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:41:17.627434 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.627447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.629779 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.629784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.631900 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.631918 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.633642 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.633649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.633841 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.633873 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 19:41:17.633942 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 19:41:17.634066 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":40,"thread_start_us":257,"threads_started":1}
I20260926 19:41:17.634763 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.626436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.2.164.67:39929)
0926 19:41:17.627453 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:17.627460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:17.627487 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:41:17.627617 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:17.627624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:17.627832 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:17.628525 (+ 693us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:41:17.628768 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.630097 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.632328 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.633355 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.634040 (+ 685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.634266 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:17.634329 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 19:41:17.634471 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":747,"threads_started":1}
I20260926 19:41:17.634805 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.625983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:39929 (local address 127.0.0.1:44386)
0926 19:41:17.626815 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 19:41:17.626833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:41:17.626946 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:41:17.628014 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:41:17.628018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:41:17.628041 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:41:17.628344 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.628352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.630225 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.630229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.632207 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.632216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.633554 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.633560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.633729 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.633763 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:41:17.633823 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 19:41:17.633929 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":3,"thread_start_us":587,"threads_started":1}
I20260926 19:41:17.635818 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.625826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.2.164.65:34743)
0926 19:41:17.626550 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:17.626559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:17.626584 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:41:17.626671 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:17.626687 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:17.626895 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:17.628100 (+ 1205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:41:17.628300 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.629662 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.632067 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.633402 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.635068 (+ 1666us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.635283 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:17.635349 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 19:41:17.635496 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":422,"threads_started":1}
I20260926 19:41:17.638322 14135 tablet_service.cc:1467] Processing CreateTablet for tablet b21b8f2ee9024db3ad639d0297ccdea8 (DEFAULT_TABLE table=test-table [id=9297c06a245443bf886d13eaf798bf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:41:17.638223 14278 tablet_service.cc:1467] Processing CreateTablet for tablet b21b8f2ee9024db3ad639d0297ccdea8 (DEFAULT_TABLE table=test-table [id=9297c06a245443bf886d13eaf798bf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:41:17.638790 14412 tablet_service.cc:1467] Processing CreateTablet for tablet b21b8f2ee9024db3ad639d0297ccdea8 (DEFAULT_TABLE table=test-table [id=9297c06a245443bf886d13eaf798bf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:41:17.639499 14135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21b8f2ee9024db3ad639d0297ccdea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:41:17.639568 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21b8f2ee9024db3ad639d0297ccdea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:41:17.639986 14412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21b8f2ee9024db3ad639d0297ccdea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:41:17.650205 14495 tablet_bootstrap.cc:492] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: Bootstrap starting.
I20260926 19:41:17.653404 14495 tablet_bootstrap.cc:654] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: Neither blocks nor log segments found. Creating new log.
I20260926 19:41:17.654376 14495 log.cc:824] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:41:17.657284 14496 tablet_bootstrap.cc:492] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: Bootstrap starting.
I20260926 19:41:17.657449 14495 tablet_bootstrap.cc:492] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: No bootstrap required, opened a new log
I20260926 19:41:17.657758 14495 ts_tablet_manager.cc:1397] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260926 19:41:17.657799 14497 tablet_bootstrap.cc:492] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: Bootstrap starting.
I20260926 19:41:17.660600 14496 tablet_bootstrap.cc:654] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: Neither blocks nor log segments found. Creating new log.
I20260926 19:41:17.660933 14497 tablet_bootstrap.cc:654] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: Neither blocks nor log segments found. Creating new log.
I20260926 19:41:17.662228 14497 log.cc:824] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: Log is configured to *not* fsync() on all Append() calls
I20260926 19:41:17.662259 14496 log.cc:824] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:41:17.664440 14497 tablet_bootstrap.cc:492] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: No bootstrap required, opened a new log
I20260926 19:41:17.664683 14497 ts_tablet_manager.cc:1397] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 19:41:17.666810 14496 tablet_bootstrap.cc:492] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: No bootstrap required, opened a new log
I20260926 19:41:17.667042 14496 ts_tablet_manager.cc:1397] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260926 19:41:17.667140 14495 raft_consensus.cc:348] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873e8178a72c4185bc5b01f7422016e5" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35227 } } peers { permanent_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34743 } } peers { permanent_uuid: "74ccfd53f595477596c73681c0a7e1f3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 39929 } }
I20260926 19:41:17.667606 14495 raft_consensus.cc:374] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:41:17.667743 14495 raft_consensus.cc:729] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdcc1a66b2e494cba2f83d2eb6067d3, State: Initialized, Role: FOLLOWER
I20260926 19:41:17.668320 14495 consensus_queue.cc:260] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873e8178a72c4185bc5b01f7422016e5" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35227 } } peers { permanent_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34743 } } peers { permanent_uuid: "74ccfd53f595477596c73681c0a7e1f3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 39929 } }
I20260926 19:41:17.669940 14495 ts_tablet_manager.cc:1428] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260926 19:41:17.673007 14497 raft_consensus.cc:348] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873e8178a72c4185bc5b01f7422016e5" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35227 } } peers { permanent_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34743 } } peers { permanent_uuid: "74ccfd53f595477596c73681c0a7e1f3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 39929 } }
I20260926 19:41:17.673434 14497 raft_consensus.cc:374] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:41:17.673532 14497 raft_consensus.cc:729] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873e8178a72c4185bc5b01f7422016e5, State: Initialized, Role: FOLLOWER
I20260926 19:41:17.674109 14497 consensus_queue.cc:260] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873e8178a72c4185bc5b01f7422016e5" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35227 } } peers { permanent_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34743 } } peers { permanent_uuid: "74ccfd53f595477596c73681c0a7e1f3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 39929 } }
I20260926 19:41:17.675072 14343 heartbeater.cc:503] Master 127.2.164.126:46059 was elected leader, sending a full tablet report...
I20260926 19:41:17.675680 14497 ts_tablet_manager.cc:1428] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 19:41:17.676723 14496 raft_consensus.cc:348] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873e8178a72c4185bc5b01f7422016e5" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35227 } } peers { permanent_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34743 } } peers { permanent_uuid: "74ccfd53f595477596c73681c0a7e1f3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 39929 } }
I20260926 19:41:17.677112 14496 raft_consensus.cc:374] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:41:17.677260 14496 raft_consensus.cc:729] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ccfd53f595477596c73681c0a7e1f3, State: Initialized, Role: FOLLOWER
I20260926 19:41:17.677774 14496 consensus_queue.cc:260] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873e8178a72c4185bc5b01f7422016e5" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35227 } } peers { permanent_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34743 } } peers { permanent_uuid: "74ccfd53f595477596c73681c0a7e1f3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 39929 } }
I20260926 19:41:17.678651 14477 heartbeater.cc:503] Master 127.2.164.126:46059 was elected leader, sending a full tablet report...
I20260926 19:41:17.679251 14496 ts_tablet_manager.cc:1428] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260926 19:41:17.770310 14344 tablet.cc:2369] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:41:17.781427 14201 tablet.cc:2369] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:41:17.812755 14478 tablet.cc:2369] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:41:17.909654 14503 raft_consensus.cc:482] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:41:17.909930 14503 raft_consensus.cc:504] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873e8178a72c4185bc5b01f7422016e5" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35227 } } peers { permanent_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34743 } } peers { permanent_uuid: "74ccfd53f595477596c73681c0a7e1f3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 39929 } }
I20260926 19:41:17.911535 14503 leader_election.cc:288] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873e8178a72c4185bc5b01f7422016e5 (127.2.164.66:35227), dcdcc1a66b2e494cba2f83d2eb6067d3 (127.2.164.65:34743)
I20260926 19:41:17.917428 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.911922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:35227 (local address 127.2.164.67:60149)
0926 19:41:17.912086 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 19:41:17.912110 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:41:17.912227 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:41:17.912629 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:41:17.912636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:41:17.912670 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:41:17.912925 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.912940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.914176 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.914180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.915465 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.915476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.916491 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.916494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.916602 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.916624 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:41:17.916687 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 19:41:17.916786 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 19:41:17.917620 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.912000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:60149 (local address 127.2.164.66:35227)
0926 19:41:17.912162 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:17.912167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:17.912185 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:41:17.912375 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:17.912381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:17.912468 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:17.912555 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:41:17.913072 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.914062 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.915618 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.916374 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.916522 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.916594 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:17.916726 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 19:41:17.916848 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 19:41:17.918581 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.912481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:44549 (local address 127.2.164.65:34743)
0926 19:41:17.912689 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:17.912695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:17.913223 (+ 528us) server_negotiation.cc:407] Connection header received
0926 19:41:17.913498 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:17.913504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:17.913577 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:17.913665 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:41:17.914198 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.914907 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.916460 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.917701 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.917861 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.917929 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:17.918215 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 19:41:17.918336 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:41:17.918604 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.912345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:34743 (local address 127.2.164.67:44549)
0926 19:41:17.913114 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 19:41:17.913131 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:41:17.913319 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:41:17.913681 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:41:17.913685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:41:17.913710 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:41:17.914019 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.914031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.915067 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.915070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.916318 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:41:17.916327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.917940 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.917952 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:41:17.918082 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.918104 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:41:17.918177 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 19:41:17.918301 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":455,"threads_started":1}
I20260926 19:41:17.918459 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b8f2ee9024db3ad639d0297ccdea8" candidate_uuid: "74ccfd53f595477596c73681c0a7e1f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873e8178a72c4185bc5b01f7422016e5" is_pre_election: true
I20260926 19:41:17.918989 14298 raft_consensus.cc:2454] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ccfd53f595477596c73681c0a7e1f3 in term 0.
I20260926 19:41:17.919412 14155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b8f2ee9024db3ad639d0297ccdea8" candidate_uuid: "74ccfd53f595477596c73681c0a7e1f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" is_pre_election: true
I20260926 19:41:17.919577 14366 leader_election.cc:302] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74ccfd53f595477596c73681c0a7e1f3, 873e8178a72c4185bc5b01f7422016e5; no voters:
I20260926 19:41:17.919842 14155 raft_consensus.cc:2454] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ccfd53f595477596c73681c0a7e1f3 in term 0.
I20260926 19:41:17.919914 14503 raft_consensus.cc:2781] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:41:17.920037 14503 raft_consensus.cc:482] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:41:17.920130 14503 raft_consensus.cc:3037] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:41:17.922219 14503 raft_consensus.cc:504] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873e8178a72c4185bc5b01f7422016e5" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35227 } } peers { permanent_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34743 } } peers { permanent_uuid: "74ccfd53f595477596c73681c0a7e1f3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 39929 } }
I20260926 19:41:17.922694 14503 leader_election.cc:288] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [CANDIDATE]: Term 1 election: Requested vote from peers 873e8178a72c4185bc5b01f7422016e5 (127.2.164.66:35227), dcdcc1a66b2e494cba2f83d2eb6067d3 (127.2.164.65:34743)
I20260926 19:41:17.923067 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b8f2ee9024db3ad639d0297ccdea8" candidate_uuid: "74ccfd53f595477596c73681c0a7e1f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873e8178a72c4185bc5b01f7422016e5"
I20260926 19:41:17.923128 14155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b8f2ee9024db3ad639d0297ccdea8" candidate_uuid: "74ccfd53f595477596c73681c0a7e1f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3"
I20260926 19:41:17.923218 14298 raft_consensus.cc:3037] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:41:17.923308 14155 raft_consensus.cc:3037] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:41:17.925269 14298 raft_consensus.cc:2454] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ccfd53f595477596c73681c0a7e1f3 in term 1.
I20260926 19:41:17.925349 14155 raft_consensus.cc:2454] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ccfd53f595477596c73681c0a7e1f3 in term 1.
I20260926 19:41:17.925585 14366 leader_election.cc:302] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74ccfd53f595477596c73681c0a7e1f3, 873e8178a72c4185bc5b01f7422016e5; no voters:
I20260926 19:41:17.925829 14503 raft_consensus.cc:2781] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:41:17.926566 14503 raft_consensus.cc:686] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 1 LEADER]: Becoming Leader. State: Replica: 74ccfd53f595477596c73681c0a7e1f3, State: Running, Role: LEADER
I20260926 19:41:17.926973 14503 consensus_queue.cc:237] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873e8178a72c4185bc5b01f7422016e5" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35227 } } peers { permanent_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34743 } } peers { permanent_uuid: "74ccfd53f595477596c73681c0a7e1f3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 39929 } }
I20260926 19:41:17.931334 13987 catalog_manager.cc:5495] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74ccfd53f595477596c73681c0a7e1f3 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "74ccfd53f595477596c73681c0a7e1f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873e8178a72c4185bc5b01f7422016e5" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74ccfd53f595477596c73681c0a7e1f3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 39929 } health_report { overall_health: HEALTHY } } }
I20260926 19:41:17.996732 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:17.991512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.2.164.67:39929)
0926 19:41:17.991703 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:17.991709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:17.991768 (+ 59us) server_negotiation.cc:407] Connection header received
0926 19:41:17.991967 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:17.991971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:17.992046 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:17.992164 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:41:17.992492 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.993263 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.994638 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:17.995050 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:17.995165 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:17.995616 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:41:17.996131 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:41:17.996345 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:17.996398 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 19:41:17.996487 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:41:18.004492 14155 raft_consensus.cc:1261] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3 [term 1 FOLLOWER]: Refusing update from remote peer 74ccfd53f595477596c73681c0a7e1f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:41:18.004492 14298 raft_consensus.cc:1261] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5 [term 1 FOLLOWER]: Refusing update from remote peer 74ccfd53f595477596c73681c0a7e1f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:41:18.005467 14505 consensus_queue.cc:1035] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcdcc1a66b2e494cba2f83d2eb6067d3" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 34743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:41:18.006304 14503 consensus_queue.cc:1035] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873e8178a72c4185bc5b01f7422016e5" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:41:18.017243 14508 mvcc.cc:205] Tried to move back new op lower bound from 7333690073096970240 to 7333690072792301568. Current Snapshot: MvccSnapshot[applied={T|T < 7333690073096970240}]
I20260926 19:41:18.027127 14510 mvcc.cc:205] Tried to move back new op lower bound from 7333690073096970240 to 7333690072792301568. Current Snapshot: MvccSnapshot[applied={T|T < 7333690073096970240}]
I20260926 19:41:18.033062 13986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:41:18.033437 13986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:41:18.037372 13986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6463065f452c4f518f0940f1ddf6b106: Sending DeleteTablet for 3 replicas of tablet b21b8f2ee9024db3ad639d0297ccdea8
I20260926 19:41:18.038313 14278 tablet_service.cc:1514] Processing DeleteTablet for tablet b21b8f2ee9024db3ad639d0297ccdea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:41:18 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:36564
I20260926 19:41:18.038436 14412 tablet_service.cc:1514] Processing DeleteTablet for tablet b21b8f2ee9024db3ad639d0297ccdea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:41:18 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:44386
I20260926 19:41:18.039624 14135 tablet_service.cc:1514] Processing DeleteTablet for tablet b21b8f2ee9024db3ad639d0297ccdea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:41:18 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:50594
I20260926 19:41:18.040763 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 14067
I20260926 19:41:18.051947 14519 tablet_replica.cc:330] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: stopping tablet replica
I20260926 19:41:18.052405 14519 raft_consensus.cc:2229] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:41:18.052942 14519 raft_consensus.cc:2258] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:41:18.055244 14519 ts_tablet_manager.cc:1905] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:41:18.058773 14518 tablet_replica.cc:330] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: stopping tablet replica
I20260926 19:41:18.058956 14517 tablet_replica.cc:330] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: stopping tablet replica
I20260926 19:41:18.059350 14518 raft_consensus.cc:2229] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:41:18.059356 14517 raft_consensus.cc:2229] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:41:18.060315 14518 raft_consensus.cc:2258] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:41:18.060304 14517 raft_consensus.cc:2258] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:41:18.062431 14517 ts_tablet_manager.cc:1905] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:41:18.063015 14519 ts_tablet_manager.cc:1918] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:41:18.063165 14519 log.cc:1196] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/b21b8f2ee9024db3ad639d0297ccdea8
I20260926 19:41:18.063604 14519 ts_tablet_manager.cc:1939] T b21b8f2ee9024db3ad639d0297ccdea8 P dcdcc1a66b2e494cba2f83d2eb6067d3: Deleting consensus metadata
I20260926 19:41:18.065502 13972 catalog_manager.cc:4841] TS dcdcc1a66b2e494cba2f83d2eb6067d3 (127.2.164.65:34743): tablet b21b8f2ee9024db3ad639d0297ccdea8 (table test-table [id=9297c06a245443bf886d13eaf798bf16]) successfully deleted
Sep 26 19:41:18 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:41:18.067688 14518 ts_tablet_manager.cc:1905] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:41:18.067768 14517 ts_tablet_manager.cc:1918] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:41:18.067902 14517 log.cc:1196] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/b21b8f2ee9024db3ad639d0297ccdea8
I20260926 19:41:18.068265 14517 ts_tablet_manager.cc:1939] T b21b8f2ee9024db3ad639d0297ccdea8 P 873e8178a72c4185bc5b01f7422016e5: Deleting consensus metadata
I20260926 19:41:18.069657 13972 catalog_manager.cc:4841] TS 873e8178a72c4185bc5b01f7422016e5 (127.2.164.66:35227): tablet b21b8f2ee9024db3ad639d0297ccdea8 (table test-table [id=9297c06a245443bf886d13eaf798bf16]) successfully deleted
I20260926 19:41:18.074622 14518 ts_tablet_manager.cc:1918] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:41:18.074780 14518 log.cc:1196] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/b21b8f2ee9024db3ad639d0297ccdea8
I20260926 19:41:18.075704 14518 ts_tablet_manager.cc:1939] T b21b8f2ee9024db3ad639d0297ccdea8 P 74ccfd53f595477596c73681c0a7e1f3: Deleting consensus metadata
I20260926 19:41:18.076261 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:18.041814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.2.164.65:34743)
0926 19:41:18.041985 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:18.041992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:18.046945 (+ 4953us) server_negotiation.cc:407] Connection header received
0926 19:41:18.047731 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:18.047737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:18.047822 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:18.047894 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:41:18.053598 (+ 5704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:18.054699 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:18.060481 (+ 5782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:18.061114 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:18.061367 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:18.070715 (+ 9348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:41:18.070726 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:41:18.070739 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:41:18.070775 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:41:18.073463 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:18.074120 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:18.074127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:18.074129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:18.074254 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:18.074622 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:18.074625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:18.074627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:18.075047 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:41:18.075331 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:18.075661 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 19:41:18.075902 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 19:41:18.076923 14195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:41:18.077359 13972 catalog_manager.cc:4841] TS 74ccfd53f595477596c73681c0a7e1f3 (127.2.164.67:39929): tablet b21b8f2ee9024db3ad639d0297ccdea8 (table test-table [id=9297c06a245443bf886d13eaf798bf16]) successfully deleted
I20260926 19:41:18.165091 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 14067
I20260926 19:41:18.185880 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 14206
Sep 26 19:41:18 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:41:18.202770 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:18.186662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.2.164.66:35227)
0926 19:41:18.186835 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:18.186839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:18.186857 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:41:18.187063 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:18.187068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:18.187136 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:18.187261 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:41:18.188820 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:18.189595 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:18.191395 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:18.191823 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:18.191943 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:18.197278 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:41:18.197294 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:41:18.197311 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:41:18.197360 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:41:18.200048 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:18.200843 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:18.200851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:18.200853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:18.200975 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:18.201453 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:18.201457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:18.201459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:18.201801 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:41:18.202005 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:18.202354 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 19:41:18.202509 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:41:18.203462 14338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:41:18.288964 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 14206
I20260926 19:41:18.306133 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 14349
Sep 26 19:41:18 dist-test-slave-mrw4 krb5kdc[13511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:41:18.325500 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:41:18.307034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.2.164.67:39929)
0926 19:41:18.307212 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:41:18.307218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:41:18.307436 (+ 218us) server_negotiation.cc:407] Connection header received
0926 19:41:18.307617 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:41:18.307623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:41:18.307717 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:41:18.307847 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:41:18.309483 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:18.310652 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:18.312498 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:41:18.313362 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:41:18.313504 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:41:18.318627 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:41:18.318642 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:41:18.318666 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:41:18.318716 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:41:18.322592 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:18.323377 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:18.323386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:18.323390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:18.323579 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:41:18.323900 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:41:18.323905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:41:18.323909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:41:18.324364 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:41:18.324697 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 19:41:18.324842 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 19:41:18.325061 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:41:18.326256 14472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:41:18.442904 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 14349
I20260926 19:41:18.458128 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 13955
I20260926 19:41:18.459242 14017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:41:18.566659 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 13955
I20260926 19:41:18.583724 2705 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 19:41:18.918304 2705 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 19:41:18.918524 2705 mini_ranger.cc:66] Stopping Ranger...
I20260926 19:41:19.253187 2705 mini_ranger.cc:68] Stopped Ranger
2026-09-26 19:41:19.253 UTC [13541] LOG: received smart shutdown request
2026-09-26 19:41:19.256 UTC [13541] LOG: background worker "logical replication launcher" (PID 13550) exited with exit code 1
2026-09-26 19:41:19.256 UTC [13545] LOG: shutting down
2026-09-26 19:41:19.407 UTC [13541] LOG: database system is shut down
2026-09-26T19:41:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46860 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:41:19 dist-test-slave-mrw4 krb5kdc[14533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:41:19 dist-test-slave-mrw4 krb5kdc[14533](info): set up 2 sockets
Sep 26 19:41:19 dist-test-slave-mrw4 krb5kdc[14533](info): commencing operation
krb5kdc: starting...
W20260926 19:41:20.533284 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:41:20 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:41:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:41:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/rangeradmin_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/rangeradmin_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/rangerlookup_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/rangerlookup_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
I20260926 19:41:20.813446 2705 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDi4fXZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/postgres -l logfile start
2026-09-26 19:41:23.154 UTC [14563] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 19:41:23.154 UTC [14563] LOG: listening on IPv4 address "127.2.164.84", port 35965
2026-09-26 19:41:23.158 UTC [14563] LOG: listening on Unix socket "/tmp/.s.PGSQL.35965"
2026-09-26 19:41:23.177 UTC [14566] LOG: database system was shut down at 2026-09-26 19:41:21 UTC
2026-09-26 19:41:23.181 UTC [14563] LOG: database system is ready to accept connections
I20260926 19:41:24.064975 2705 mini_postgres.cc:96] Postgres bound to 35965
2026-09-26 19:41:24.080 UTC [14574] FATAL: database "slave" does not exist
127.2.164.84:35965 - accepting connections
I20260926 19:41:24.081260 2705 mini_ranger.cc:161] Starting Ranger...
I20260926 19:41:24.104557 2705 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 19:41:24.541867 2705 mini_ranger.cc:87] Created ranger Postgres database
I20260926 19:41:24.541958 2705 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-admin
2026-09-26 19:41:24,814 [I] DB FLAVOR :POSTGRES
2026-09-26 19:41:24,814 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 19:41:24,814 [I] Checking connection..
2026-09-26 19:41:24,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 19:41:25,056 [I] Checking connection passed.
2026-09-26 19:41:25,056 [I] --------- Verifying version history table ---------
2026-09-26 19:41:25,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:41:25,298 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 19:41:25,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:41:25,548 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 19:41:25,549 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 19:41:25,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 19:41:25.780 UTC [14638] WARNING: there is no transaction in progress
2026-09-26 19:41:25,786 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 19:41:25,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:41:26,012 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 19:41:26,012 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 19:41:26,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 19:41:26,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 19:41:26,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mrw4.c.gcp-upstream.internal','N') ;"
2026-09-26 19:41:26,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 19:41:26,947 [I] Table x_portal_user does not exist in database ranger
2026-09-26 19:41:26,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 19:41:27,167 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 19:41:27,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 19:41:27,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 19:41:27,652 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 19:41:27,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 19:41:28.560 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.575 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.590 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.605 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.620 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.638 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.699 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.706 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.720 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.729 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.738 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.751 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.760 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.766 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.772 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:28.778 UTC [14746] WARNING: there is no transaction in progress
2026-09-26 19:41:29,386 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 19:41:29,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 19:41:29,611 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 19:41:29,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 19:41:29,860 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 19:41:29,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 19:41:30,103 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 19:41:30,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 19:41:30,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 19:41:30,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mrw4.c.gcp-upstream.internal';"
2026-09-26 19:41:30,781 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 19:41:30,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 19:41:31,020 [I] DB_PATCHES have already been applied
I20260926 19:41:31.026062 2705 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDi4fXZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 19:41:31.026145 2705 mini_ranger.cc:193] Using host: 127.2.164.84
I20260926 19:41:31.038935 2705 mini_ranger.cc:227] Ranger admin URL: http://127.2.164.84:47881
Sep 26, 2026 7:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:41:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 19:41:32.031023 2705 mini_ranger.cc:160] Time spent starting Ranger: real 7.950s user 0.001s sys 0.034s
Sep 26, 2026 7:41:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:41:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 7:41:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:41:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:41:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 19:41:52.827700 2705 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.164.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.164.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/rangerkms_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.164.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/rangerkms_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.164.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.164.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.164.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/HTTP_127.2.164.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 19:41:52 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451712, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 19:41:53.000589 2705 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 19:41:53.018196 2705 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 19:41:53.789980 2705 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 19:41:53.790071 2705 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-kms
I20260926 19:41:53.790108 2705 mini_ranger_kms.cc:227] Using postgres at 127.2.164.84:35965
2026-09-26 19:41:54,100 [I] DB FLAVOR :POSTGRES
2026-09-26 19:41:54,100 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 19:41:54,100 [I] Checking connection
2026-09-26 19:41:54,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 19:41:54,320 [I] connection success
2026-09-26 19:41:54,320 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 19:41:54,320 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 19:41:54,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 19:41:54,567 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 19:41:54,568 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 19:41:54,568 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 19:41:54,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.164.84:35965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 19:41:54,806 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 19:41:55.250820 2705 mini_ranger_kms.cc:310] Created kms service
I20260926 19:41:55.455704 2705 mini_ranger_kms.cc:326] Created kudu user
I20260926 19:41:55.499796 2705 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 19:41:55 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451712, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:41:55.859925 2705 mini_ranger_kms.cc:384] Added ranger policy
I20260926 19:41:55.860244 2705 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDi4fXZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDi4fXZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 19:41:55.860301 2705 mini_ranger_kms.cc:240] Using host: 127.2.164.84
I20260926 19:41:55.874007 2705 mini_ranger_kms.cc:276] Ranger KMS PID: 14926
I20260926 19:41:55.874119 2705 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.164.84:46971
Sep 26, 2026 7:41:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:41:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:41:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:41:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:41:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:41:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 19:41:57 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790451717, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:41:57.901281 2705 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.901s user 0.043s sys 0.276s
I20260926 19:41:57.901508 2705 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 19:41:57.901670 2705 mini_ranger_kms.cc:401] 127.2.164.84:46971/kms/v1/keys
Sep 26 19:41:58 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451717, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
Sep 26 19:41:58 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451717, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.164.84@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
Sep 26 19:42:00 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:42:00.873858 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40225
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:40225
--ranger_config_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:46971/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:01.178324 14977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:01.178608 14977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:01.178732 14977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:01.185839 14977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:42:01.185950 14977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:01.186023 14977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:01.186093 14977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:42:01.186154 14977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:42:01.196005 14977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.164.84:46971/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:40225
--ranger_config_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:01.197263 14977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:01.198544 14977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:01.210696 14982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:01.210738 14983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:01.211043 14985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:01.211369 14977 server_base.cc:1034] running on GCE node
I20260926 19:42:01.212127 14977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:01.213565 14977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:01.214776 14977 hybrid_clock.cc:648] HybridClock initialized: now 1790451721214730 us; error 42 us; skew 500 ppm
I20260926 19:42:01.215093 14977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:01 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451721, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:01.221230 14977 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:42:01.222915 14977 webserver.cc:460] Webserver started at http://127.2.164.126:43219/ using document root <none> and password file <none>
I20260926 19:42:01.223726 14977 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:01.223871 14977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:01.224195 14977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:42:01 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451721, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:42:01.272861 14977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "fa193ca8ee4c4ba3b75c1ecd40476911"
format_stamp: "Formatted at 2026-09-26 19:42:01 on dist-test-slave-mrw4"
server_key: "c1e403e4acdb1c6d4395368e86fa58cf"
server_key_iv: "3fd318897c42814f8d1113e00f61fcdc"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:01.274073 14977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "fa193ca8ee4c4ba3b75c1ecd40476911"
format_stamp: "Formatted at 2026-09-26 19:42:01 on dist-test-slave-mrw4"
server_key: "c1e403e4acdb1c6d4395368e86fa58cf"
server_key_iv: "3fd318897c42814f8d1113e00f61fcdc"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:01.280937 14977 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 19:42:01.301008 14992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:01.302716 14977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 19:42:01.302917 14977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "fa193ca8ee4c4ba3b75c1ecd40476911"
format_stamp: "Formatted at 2026-09-26 19:42:01 on dist-test-slave-mrw4"
server_key: "c1e403e4acdb1c6d4395368e86fa58cf"
server_key_iv: "3fd318897c42814f8d1113e00f61fcdc"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:01.303162 14977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:01.347556 14977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:01.352519 14977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:01.352842 14977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:01.370556 15043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:40225 every 8 connection(s)
I20260926 19:42:01.370568 14977 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:40225
I20260926 19:42:01.372380 14977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:42:01.374948 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 14977
I20260926 19:42:01.375142 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:42:01.375958 2705 external_mini_cluster.cc:1428] Setting key ebce29ce86f1364769bf1ca4acd072e5
I20260926 19:42:01.376749 15044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:01.391662 15044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911: Bootstrap starting.
I20260926 19:42:01.395089 15044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:01.396082 15044 log.cc:824] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:01.399094 15044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911: No bootstrap required, opened a new log
Sep 26 19:42:01 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:01.409444 15044 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } }
I20260926 19:42:01.409888 15044 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:01.410027 15044 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa193ca8ee4c4ba3b75c1ecd40476911, State: Initialized, Role: FOLLOWER
I20260926 19:42:01.410085 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:01.377971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.2.164.126:40225)
0926 19:42:01.389636 (+ 11665us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:01.389653 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:01.390060 (+ 407us) server_negotiation.cc:407] Connection header received
0926 19:42:01.392444 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:01.392472 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:01.393008 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:01.393463 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:01.395042 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:01.397028 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:01.398321 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:01.398957 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:01.399129 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:01.402579 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:01.402586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:01.402599 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:01.402635 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:01.405909 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:01.406536 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:01.406542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:01.406547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:01.406700 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:01.407050 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:01.407054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:01.407055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:01.407673 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:01.407971 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:01.408458 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 19:42:01.408853 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11437,"thread_start_us":11229,"threads_started":1}
I20260926 19:42:01.410902 15044 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } }
I20260926 19:42:01.411176 15044 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:42:01.411347 15044 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:42:01.411549 15044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:01.414703 15044 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } }
I20260926 19:42:01.415258 15044 leader_election.cc:302] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa193ca8ee4c4ba3b75c1ecd40476911; no voters:
I20260926 19:42:01.415869 15044 leader_election.cc:288] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:42:01.416115 15049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:42:01.417418 15049 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 1 LEADER]: Becoming Leader. State: Replica: fa193ca8ee4c4ba3b75c1ecd40476911, State: Running, Role: LEADER
I20260926 19:42:01.418005 15044 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:42:01.417968 15049 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } }
I20260926 19:42:01.420292 15051 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa193ca8ee4c4ba3b75c1ecd40476911. Latest consensus state: current_term: 1 leader_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } } }
I20260926 19:42:01.420606 15051 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:01.421111 15050 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } } }
I20260926 19:42:01.421365 15050 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:01.432328 15054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:42:01.437901 15054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:42:01.456328 15044 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 19:42:01.461825 15054 catalog_manager.cc:1347] Generated new cluster ID: 8273973dd89840b581e65c1335d4dc07
I20260926 19:42:01.461927 15054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:42:01.481737 15054 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:42:01.483664 15054 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:42:01.506186 15054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911: Generated new TSK 0
I20260926 19:42:01.507470 15054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:42:02.272864 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:46971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40225
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:42:02 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790451722, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 19:42:02.672971 15089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:02.673357 15089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:02.673507 15089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:02.680900 15089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:02.681007 15089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:02.681200 15089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:42:02.691383 15089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.164.84:46971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:02.692803 15089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:02.694085 15089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:02.708693 15099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:02.710439 15100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:02.717374 15102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:02.718219 15089 server_base.cc:1034] running on GCE node
I20260926 19:42:02.718937 15089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:02.720036 15089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:02.721264 15089 hybrid_clock.cc:648] HybridClock initialized: now 1790451722721203 us; error 51 us; skew 500 ppm
I20260926 19:42:02.721565 15089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:02 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451722, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:02.724486 15089 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:42:02.729545 15089 webserver.cc:460] Webserver started at http://127.2.164.65:40125/ using document root <none> and password file <none>
I20260926 19:42:02.730244 15089 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:02.730360 15089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:02.730657 15089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:42:02 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451722, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:42:02.754026 15089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "2d71a77f3c71479ba91cb9578e5eb4e1"
format_stamp: "Formatted at 2026-09-26 19:42:02 on dist-test-slave-mrw4"
server_key: "6a4f3331661c4ad4abfd373da53cde5d"
server_key_iv: "e75a5d24dfb33ca08a83336cceb7f6c7"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:02.754899 15089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "2d71a77f3c71479ba91cb9578e5eb4e1"
format_stamp: "Formatted at 2026-09-26 19:42:02 on dist-test-slave-mrw4"
server_key: "6a4f3331661c4ad4abfd373da53cde5d"
server_key_iv: "e75a5d24dfb33ca08a83336cceb7f6c7"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:02.770784 15089 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.012s sys 0.004s
I20260926 19:42:02.808779 15109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:02.814641 15089 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.006s sys 0.009s
I20260926 19:42:02.814918 15089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "2d71a77f3c71479ba91cb9578e5eb4e1"
format_stamp: "Formatted at 2026-09-26 19:42:02 on dist-test-slave-mrw4"
server_key: "6a4f3331661c4ad4abfd373da53cde5d"
server_key_iv: "e75a5d24dfb33ca08a83336cceb7f6c7"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:02.815253 15089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:02.858273 15089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:02.863459 15089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:02.863773 15089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:02.877552 15089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:02.881374 15089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:02.881516 15089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:02.881706 15089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:02.881839 15089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:02.937702 15089 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:44779
I20260926 19:42:02.937733 15221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:44779 every 8 connection(s)
I20260926 19:42:02.939038 15089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:42:02.941859 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 15089
I20260926 19:42:02.942056 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:42:02.942875 2705 external_mini_cluster.cc:1428] Setting key 4065191b4c3660fe81d71d178f16f477
Sep 26 19:42:02 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451722, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260926 19:42:02.972193 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:02.944642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40225 (local address 127.2.164.65:34653)
0926 19:42:02.946044 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:02.946090 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:02.950474 (+ 4384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:02.952861 (+ 2387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:02.952873 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:02.953505 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:02.954473 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:02.954502 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:02.956354 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:02.956360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:02.958403 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:02.958427 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:02.959646 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:02.959654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:02.959935 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:02.960668 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:02.960698 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:02.963445 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:02.967455 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:02.967464 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:02.967481 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:02.968045 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:02.968514 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:02.968519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:02.968522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:02.968765 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:02.969510 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:02.969518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:02.969929 (+ 411us) client_negotiation.cc:769] Sending connection context
0926 19:42:02.970226 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 19:42:02.970759 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"thread_start_us":4614,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
I20260926 19:42:02.973677 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:02.945503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:34653 (local address 127.2.164.126:40225)
0926 19:42:02.951339 (+ 5836us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:02.951343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:02.951357 (+ 14us) server_negotiation.cc:407] Connection header received
0926 19:42:02.951403 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:02.951407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:02.951486 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:02.951575 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:02.954682 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:02.956166 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:02.958597 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:02.959220 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:02.959358 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:02.963706 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:02.963713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:02.963719 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:02.963770 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:02.967285 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:02.968209 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:02.968214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:02.968218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:02.968378 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:02.968903 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:02.968908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:02.968911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:02.969295 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:02.969433 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:02.972967 (+ 3534us) server_negotiation.cc:299] Negotiation successful
0926 19:42:02.973282 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5729,"thread_start_us":7050,"threads_started":1}
I20260926 19:42:02.975342 15222 heartbeater.cc:347] Connected to a master server at 127.2.164.126:40225
I20260926 19:42:02.975649 15222 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:02.976490 15222 heartbeater.cc:511] Master 127.2.164.126:40225 requested a full tablet report, sending...
I20260926 19:42:02.979043 15009 ts_manager.cc:194] Registered new tserver with Master: 2d71a77f3c71479ba91cb9578e5eb4e1 (127.2.164.65:44779)
I20260926 19:42:02.981053 15009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:34653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:42:03.005764 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:46971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40225
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:42:03 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451722, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
Sep 26 19:42:03 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451722, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
W20260926 19:42:03.421279 15228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:03.421767 15228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:03.421932 15228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:03.433677 15228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:03.433826 15228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:03.434053 15228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:42:03.449695 15228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.164.84:46971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:03.451516 15228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:03.453155 15228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:03.475472 15235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:03.476115 15236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:03.487165 15238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:03.491406 15228 server_base.cc:1034] running on GCE node
I20260926 19:42:03.492269 15228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:03.501454 15228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:03.502767 15228 hybrid_clock.cc:648] HybridClock initialized: now 1790451723502673 us; error 86 us; skew 500 ppm
I20260926 19:42:03.503212 15228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:03 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:03.507557 15228 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:42:03.528039 15228 webserver.cc:460] Webserver started at http://127.2.164.66:35493/ using document root <none> and password file <none>
I20260926 19:42:03.529246 15228 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:03.529472 15228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:03.529898 15228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:42:03 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:42:03.559873 15228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "b34d470b95bf483b9eb985f0de126961"
format_stamp: "Formatted at 2026-09-26 19:42:03 on dist-test-slave-mrw4"
server_key: "0dffb67ba791ef7fc3a66eb1cbb3c774"
server_key_iv: "ba947759686920f8cecf8ee4dbb43ae8"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:03.573951 15228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "b34d470b95bf483b9eb985f0de126961"
format_stamp: "Formatted at 2026-09-26 19:42:03 on dist-test-slave-mrw4"
server_key: "0dffb67ba791ef7fc3a66eb1cbb3c774"
server_key_iv: "ba947759686920f8cecf8ee4dbb43ae8"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:03.595233 15228 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.016s sys 0.004s
I20260926 19:42:03.633765 15245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:03.638679 15228 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.006s sys 0.003s
I20260926 19:42:03.638948 15228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "b34d470b95bf483b9eb985f0de126961"
format_stamp: "Formatted at 2026-09-26 19:42:03 on dist-test-slave-mrw4"
server_key: "0dffb67ba791ef7fc3a66eb1cbb3c774"
server_key_iv: "ba947759686920f8cecf8ee4dbb43ae8"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:03.639312 15228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:03.713544 15228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:03.718757 15228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:03.719125 15228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:03.726475 15228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:03.734727 15228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:03.734882 15228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:03.735046 15228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:03.735150 15228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:03.984737 15222 heartbeater.cc:503] Master 127.2.164.126:40225 was elected leader, sending a full tablet report...
I20260926 19:42:03.992609 15228 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:40187
I20260926 19:42:03.992648 15364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:40187 every 8 connection(s)
I20260926 19:42:03.994315 15228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:42:04.002136 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 15228
I20260926 19:42:04.002311 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:42:04.002918 2705 external_mini_cluster.cc:1428] Setting key 27d59c518dbbc555e98c449be199ed5e
Sep 26 19:42:04 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
I20260926 19:42:04.037009 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:03.997432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:52145 (local address 127.2.164.126:40225)
0926 19:42:04.019384 (+ 21952us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:04.019390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:04.019409 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:42:04.019466 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:04.019471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:04.019542 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:04.019636 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:04.021871 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.022951 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.025141 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.025758 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.025846 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:04.030197 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:04.030216 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:04.030221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:04.030264 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:04.033135 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:04.034379 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:04.034384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:04.034387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:04.034537 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:04.035034 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:04.035039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:04.035043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:04.035375 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:04.035499 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:04.036414 (+ 915us) server_negotiation.cc:299] Negotiation successful
0926 19:42:04.036631 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21824,"thread_start_us":21709,"threads_started":1}
I20260926 19:42:04.039252 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:03.998082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40225 (local address 127.2.164.66:52145)
0926 19:42:04.016303 (+ 18221us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:04.016368 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:04.018639 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:04.020077 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:04.020090 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:04.020760 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:04.021626 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:04.021653 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.023146 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.023150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:04.024986 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:04.025000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.026166 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.026172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:04.026450 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:04.027183 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:04.027229 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:04.029947 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:04.033379 (+ 3432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:04.033388 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:04.033429 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:04.033858 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:04.034683 (+ 825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:04.034688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:04.034691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:04.034914 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:04.035586 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:04.035594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:04.036025 (+ 431us) client_negotiation.cc:769] Sending connection context
0926 19:42:04.037319 (+ 1294us) client_negotiation.cc:240] Negotiation successful
0926 19:42:04.037858 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17892,"thread_start_us":17654,"threads_started":1}
I20260926 19:42:04.040894 15365 heartbeater.cc:347] Connected to a master server at 127.2.164.126:40225
I20260926 19:42:04.041234 15365 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:04.041883 15365 heartbeater.cc:511] Master 127.2.164.126:40225 requested a full tablet report, sending...
I20260926 19:42:04.043279 15009 ts_manager.cc:194] Registered new tserver with Master: b34d470b95bf483b9eb985f0de126961 (127.2.164.66:40187)
I20260926 19:42:04.044292 15009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:52145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:42:04.064649 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:46971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40225
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:04.389962 15371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:04.390240 15371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:04.390379 15371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:04.397469 15371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:04.397636 15371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:04.397794 15371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:42:04.407287 15371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.164.84:46971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:04.408608 15371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:04.409891 15371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:04.423243 15376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:04.423342 15379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:04.423273 15377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:04.423945 15371 server_base.cc:1034] running on GCE node
I20260926 19:42:04.424466 15371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:04.425336 15371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:04.426525 15371 hybrid_clock.cc:648] HybridClock initialized: now 1790451724426470 us; error 48 us; skew 500 ppm
I20260926 19:42:04.426833 15371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:04 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451724, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:04.429721 15371 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:42:04.430974 15371 webserver.cc:460] Webserver started at http://127.2.164.67:33557/ using document root <none> and password file <none>
I20260926 19:42:04.431602 15371 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:04.431710 15371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:04.431978 15371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:42:04 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451724, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:42:04.446938 15371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "acc2ec14cf0d45b98f3b15f8f8525f83"
format_stamp: "Formatted at 2026-09-26 19:42:04 on dist-test-slave-mrw4"
server_key: "1e7f8551880d04eb537f2baeb2fb7162"
server_key_iv: "202010794d3e4684175a71826c4078a7"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:04.447708 15371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "acc2ec14cf0d45b98f3b15f8f8525f83"
format_stamp: "Formatted at 2026-09-26 19:42:04 on dist-test-slave-mrw4"
server_key: "1e7f8551880d04eb537f2baeb2fb7162"
server_key_iv: "202010794d3e4684175a71826c4078a7"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:04.452986 15371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:42:04.466267 15386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:04.467653 15371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:42:04.467806 15371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "acc2ec14cf0d45b98f3b15f8f8525f83"
format_stamp: "Formatted at 2026-09-26 19:42:04 on dist-test-slave-mrw4"
server_key: "1e7f8551880d04eb537f2baeb2fb7162"
server_key_iv: "202010794d3e4684175a71826c4078a7"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:04.468004 15371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:04.492457 15371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:04.495731 15371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:04.495973 15371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:04.496853 15371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:04.498111 15371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:04.498178 15371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:04.498279 15371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:04.498337 15371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:04.527370 15371 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:34327
I20260926 19:42:04.527405 15498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:34327 every 8 connection(s)
I20260926 19:42:04.528569 15371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:42:04.529749 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 15371
I20260926 19:42:04.529922 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:42:04.530453 2705 external_mini_cluster.cc:1428] Setting key 3455af7ba2272ec17955018498d15b48
Sep 26 19:42:04 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451724, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:04.547204 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:04.531204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:43143 (local address 127.2.164.126:40225)
0926 19:42:04.531405 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:04.531411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:04.532714 (+ 1303us) server_negotiation.cc:407] Connection header received
0926 19:42:04.534554 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:04.534559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:04.534669 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:04.534750 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:04.536088 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.536787 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.538285 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.538684 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.538788 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:04.541832 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:04.541838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:04.541843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:04.541876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:04.544171 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:04.545051 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:04.545056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:04.545059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:04.545267 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:04.545779 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:04.545784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:04.545787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:04.546052 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:04.546167 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:04.546819 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 19:42:04.546986 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:42:04.548074 15501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:04.531841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40225 (local address 127.2.164.67:43143)
0926 19:42:04.532570 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:04.532611 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:04.534309 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:04.534981 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:04.534989 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:04.535393 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:04.535941 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:04.535958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.536917 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.536926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:04.538149 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:04.538167 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.538881 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.538888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:04.539104 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:04.539655 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:04.539679 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:04.541642 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:04.544331 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:04.544348 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:04.544365 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:04.544922 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:04.545396 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:04.545400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:04.545403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:04.545654 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:04.546165 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:04.546170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:04.546497 (+ 327us) client_negotiation.cc:769] Sending connection context
0926 19:42:04.546712 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 19:42:04.547050 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":284,"threads_started":1}
I20260926 19:42:04.549316 15499 heartbeater.cc:347] Connected to a master server at 127.2.164.126:40225
I20260926 19:42:04.549569 15499 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:04.550189 15499 heartbeater.cc:511] Master 127.2.164.126:40225 requested a full tablet report, sending...
I20260926 19:42:04.551565 15009 ts_manager.cc:194] Registered new tserver with Master: acc2ec14cf0d45b98f3b15f8f8525f83 (127.2.164.67:34327)
I20260926 19:42:04.552524 15009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:43143
I20260926 19:42:04.557963 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:42:04.559626 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 15089
Sep 26 19:42:04 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:42:04.576514 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:04.560421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.2.164.65:44779)
0926 19:42:04.560942 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:04.560948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:04.560971 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:42:04.561035 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:04.561041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:04.561277 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:04.561404 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:04.562645 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.563655 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.564932 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.565518 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.565611 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:04.569890 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:04.569904 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:04.569922 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:04.569964 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:04.573682 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:04.574376 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:04.574386 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:04.574390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:04.574600 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:04.574987 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:04.574992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:04.574996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:04.575320 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:04.575584 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:04.575903 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 19:42:04.576156 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":266,"threads_started":1}
I20260926 19:42:04.577761 15217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:04.658914 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 15089
I20260926 19:42:04.671355 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 15228
Sep 26 19:42:04 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:42:04.686527 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:04.672118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.2.164.66:40187)
0926 19:42:04.672662 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:04.672670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:04.672694 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:42:04.672760 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:04.672767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:04.672960 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:04.673086 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:04.674245 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.675273 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.676546 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.677064 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.677155 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:04.681378 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:04.681388 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:04.681412 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:04.681455 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:04.684254 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:04.684828 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:04.684835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:04.684837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:04.684981 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:04.685287 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:04.685290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:04.685292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:04.685619 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:04.685838 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:04.686045 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 19:42:04.686254 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":291,"threads_started":1}
I20260926 19:42:04.687779 15360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:04.766608 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 15228
I20260926 19:42:04.778853 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 15371
Sep 26 19:42:04 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:42:04.793860 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:04.779761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.2.164.67:34327)
0926 19:42:04.780404 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:04.780411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:04.780440 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:42:04.780504 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:04.780510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:04.780720 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:04.780865 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:04.782079 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.783075 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.784299 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:04.784840 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:04.784935 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:04.788956 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:04.788966 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:04.788980 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:04.789010 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:04.791518 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:04.792092 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:04.792099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:04.792101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:04.792213 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:04.792513 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:04.792516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:04.792518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:04.792823 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:04.793033 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:04.793316 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 19:42:04.793497 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":318,"threads_started":1}
I20260926 19:42:04.794989 15494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:04.874630 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 15371
I20260926 19:42:04.886735 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 14977
I20260926 19:42:04.887863 15039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:04.991034 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 14977
I20260926 19:42:05.008154 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40225
--webserver_interface=127.2.164.126
--webserver_port=43219
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:40225
--ranger_config_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:46971/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.164.84:46971/kms with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:05.310570 15510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:05.310840 15510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:05.310973 15510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:05.317827 15510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:42:05.317929 15510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:05.318003 15510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:05.318073 15510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:42:05.318135 15510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:42:05.328616 15510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.164.84:46971/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:40225
--ranger_config_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:40225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=43219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:05.329855 15510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:05.331056 15510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:05.341629 15518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:05.341784 15515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:05.341588 15516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:05.342485 15510 server_base.cc:1034] running on GCE node
I20260926 19:42:05.343111 15510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:05.344334 15510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:05.345530 15510 hybrid_clock.cc:648] HybridClock initialized: now 1790451725345486 us; error 31 us; skew 500 ppm
I20260926 19:42:05.345856 15510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:05 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451725, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:05.348654 15510 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:42:05.350023 15510 webserver.cc:460] Webserver started at http://127.2.164.126:43219/ using document root <none> and password file <none>
I20260926 19:42:05.350656 15510 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:05.350768 15510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 19:42:05 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451725, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:42:05.414623 15510 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 19:42:05.418165 15526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:05.419793 15510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:42:05.419981 15510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "fa193ca8ee4c4ba3b75c1ecd40476911"
format_stamp: "Formatted at 2026-09-26 19:42:01 on dist-test-slave-mrw4"
server_key: "c1e403e4acdb1c6d4395368e86fa58cf"
server_key_iv: "3fd318897c42814f8d1113e00f61fcdc"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:05.421286 15510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:05.435752 15510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:05.439121 15510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:05.439345 15510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:05.458034 15510 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:40225
I20260926 19:42:05.458060 15577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:40225 every 8 connection(s)
I20260926 19:42:05.459775 15510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:42:05.466400 15578 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:42:05.467730 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 15510
I20260926 19:42:05.468729 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:46971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:44779
--local_ip_for_outbound_sockets=127.2.164.65
--tserver_master_addrs=127.2.164.126:40225
--webserver_port=40125
--webserver_interface=127.2.164.65
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.164.84:46971/kms with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:42:05.470403 15578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911: Bootstrap starting.
I20260926 19:42:05.489440 15578 log.cc:824] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:05.497730 15578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:42:05.498175 15578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911: Bootstrap complete.
I20260926 19:42:05.506644 15578 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } }
I20260926 19:42:05.507469 15578 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa193ca8ee4c4ba3b75c1ecd40476911, State: Initialized, Role: FOLLOWER
I20260926 19:42:05.508054 15578 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } }
I20260926 19:42:05.508268 15578 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:42:05.508376 15578 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:42:05.508531 15578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:42:05.512277 15578 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } }
I20260926 19:42:05.512727 15578 leader_election.cc:302] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa193ca8ee4c4ba3b75c1ecd40476911; no voters:
I20260926 19:42:05.513312 15578 leader_election.cc:288] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 19:42:05.513465 15582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 19:42:05.514523 15582 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [term 2 LEADER]: Becoming Leader. State: Replica: fa193ca8ee4c4ba3b75c1ecd40476911, State: Running, Role: LEADER
I20260926 19:42:05.515105 15582 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } }
I20260926 19:42:05.515369 15578 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:42:05.517544 15583 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } } }
I20260926 19:42:05.517902 15583 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:05.520107 15584 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa193ca8ee4c4ba3b75c1ecd40476911. Latest consensus state: current_term: 2 leader_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa193ca8ee4c4ba3b75c1ecd40476911" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 40225 } } }
I20260926 19:42:05.520428 15584 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:05.521579 15588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:42:05.527756 15588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:42:05.529273 15588 catalog_manager.cc:1259] Loaded cluster ID: 8273973dd89840b581e65c1335d4dc07
I20260926 19:42:05.529421 15588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:42:05.533530 15588 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:42:05.535104 15588 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911: Loaded TSK: 0
I20260926 19:42:05.536711 15588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:42:05.542639 15578 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 19:42:05.886776 15580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:05.887218 15580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:05.887410 15580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:05.898559 15580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:05.898697 15580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:05.898887 15580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:42:05.913915 15580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.164.84:46971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:44779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=40125
--tserver_master_addrs=127.2.164.126:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:05.915570 15580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:05.917311 15580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:05.968075 15616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:05.970106 15615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:05.972167 15580 server_base.cc:1034] running on GCE node
W20260926 19:42:05.973659 15618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:05.974537 15580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:05.979290 15580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:05.980587 15580 hybrid_clock.cc:648] HybridClock initialized: now 1790451725980509 us; error 66 us; skew 500 ppm
I20260926 19:42:05.981001 15580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:05 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451725, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:05.987257 15580 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:42:06.002799 15580 webserver.cc:460] Webserver started at http://127.2.164.65:40125/ using document root <none> and password file <none>
I20260926 19:42:06.003824 15580 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:06.004033 15580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 19:42:06 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451725, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:42:06.043006 15580 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.009s sys 0.000s
I20260926 19:42:06.050665 15627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:06.052804 15580 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.007s sys 0.000s
I20260926 19:42:06.053018 15580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "2d71a77f3c71479ba91cb9578e5eb4e1"
format_stamp: "Formatted at 2026-09-26 19:42:02 on dist-test-slave-mrw4"
server_key: "6a4f3331661c4ad4abfd373da53cde5d"
server_key_iv: "e75a5d24dfb33ca08a83336cceb7f6c7"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:06.054533 15580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:06.081816 15580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:06.086323 15580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:06.086599 15580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:06.087630 15580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:06.089399 15580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:06.089506 15580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:06.089613 15580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:06.089694 15580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:06.143868 15580 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:44779
I20260926 19:42:06.143944 15739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:44779 every 8 connection(s)
I20260926 19:42:06.145661 15580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:42:06.155900 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 15580
I20260926 19:42:06.156724 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:46971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:40187
--local_ip_for_outbound_sockets=127.2.164.66
--tserver_master_addrs=127.2.164.126:40225
--webserver_port=35493
--webserver_interface=127.2.164.66
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.164.84:46971/kms with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:42:06 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451725, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:06.176625 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:06.150989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40225 (local address 127.2.164.65:55403)
0926 19:42:06.154362 (+ 3373us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:06.154423 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:06.156521 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:06.158583 (+ 2062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:06.158594 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:06.159220 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:06.159803 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:06.159825 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:06.161567 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:06.161572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:06.163088 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:06.163101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:06.164085 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:06.164092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:06.164335 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:06.165017 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:06.165053 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:06.167689 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:06.171733 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:06.171742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:06.171757 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:06.172356 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:06.172873 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:06.172877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:06.172881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:06.173120 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:06.174080 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:06.174088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:06.174487 (+ 399us) client_negotiation.cc:769] Sending connection context
0926 19:42:06.174760 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 19:42:06.175288 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3028,"thread_start_us":2757,"threads_started":1}
I20260926 19:42:06.176683 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:06.152096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:55403 (local address 127.2.164.126:40225)
0926 19:42:06.157257 (+ 5161us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:06.157278 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:06.157325 (+ 47us) server_negotiation.cc:407] Connection header received
0926 19:42:06.157743 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:06.157761 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:06.158093 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:06.158419 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:06.160005 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:06.161415 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:06.163228 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:06.163868 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:06.164031 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:06.167961 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:06.167971 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:06.167988 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:06.168029 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:06.171576 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:06.172534 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:06.172544 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:06.172553 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:06.172738 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:06.173259 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:06.173265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:06.173268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:06.173934 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:06.174139 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:06.175029 (+ 890us) server_negotiation.cc:299] Negotiation successful
0926 19:42:06.175443 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3163,"thread_start_us":2956,"threads_started":1}
I20260926 19:42:06.180194 15740 heartbeater.cc:347] Connected to a master server at 127.2.164.126:40225
I20260926 19:42:06.180528 15740 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:06.181439 15740 heartbeater.cc:511] Master 127.2.164.126:40225 requested a full tablet report, sending...
W20260926 19:42:06.184875 15740 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:40225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 19:42:06.185117 15740 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:06.185422 15740 heartbeater.cc:511] Master 127.2.164.126:40225 requested a full tablet report, sending...
I20260926 19:42:06.186717 15740 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:06.187116 15740 heartbeater.cc:511] Master 127.2.164.126:40225 requested a full tablet report, sending...
W20260926 19:42:06.188271 15740 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 19:42:06.576825 15744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:06.577086 15744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:06.577234 15744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:06.585813 15744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:06.585932 15744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:06.586098 15744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:42:06.599813 15744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.164.84:46971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:40187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=35493
--tserver_master_addrs=127.2.164.126:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:06.601619 15744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:06.603267 15744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:06.626991 15761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:06.628059 15758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:06.628279 15763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:06.629408 15744 server_base.cc:1034] running on GCE node
I20260926 19:42:06.630128 15744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:06.649468 15744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:06.650821 15744 hybrid_clock.cc:648] HybridClock initialized: now 1790451726650735 us; error 79 us; skew 500 ppm
I20260926 19:42:06.651283 15744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:06 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451726, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:06.655505 15744 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:42:06.662724 15744 webserver.cc:460] Webserver started at http://127.2.164.66:35493/ using document root <none> and password file <none>
I20260926 19:42:06.663671 15744 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:06.663826 15744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 19:42:06 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451726, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:42:06.696049 15744 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:42:06.706117 15771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Sep 26 19:42:06 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790451726, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:06.708658 15744 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.008s
I20260926 19:42:06.708840 15744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "b34d470b95bf483b9eb985f0de126961"
format_stamp: "Formatted at 2026-09-26 19:42:03 on dist-test-slave-mrw4"
server_key: "0dffb67ba791ef7fc3a66eb1cbb3c774"
server_key_iv: "ba947759686920f8cecf8ee4dbb43ae8"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:06.710278 15744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:06.747303 15744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:06.751240 15744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:06.751494 15744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:06.752413 15744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:06.753718 15744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:06.753801 15744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:06.753899 15744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:06.753962 15744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:06.805001 15744 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:40187
I20260926 19:42:06.805081 15884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:40187 every 8 connection(s)
I20260926 19:42:06.806869 15744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:42:06.807467 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 15744
I20260926 19:42:06.808238 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.164.84:46971/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:34327
--local_ip_for_outbound_sockets=127.2.164.67
--tserver_master_addrs=127.2.164.126:40225
--webserver_port=33557
--webserver_interface=127.2.164.67
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.164.84:46971/kms with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:42:06 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451726, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:06.837044 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:06.810462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40225 (local address 127.2.164.66:42697)
0926 19:42:06.817670 (+ 7208us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:06.817723 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:06.820307 (+ 2584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:06.821571 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:06.821579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:06.821998 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:06.822639 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:06.822667 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:06.823898 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:06.823903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:06.825983 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:06.825997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:06.827300 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:06.827304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:06.827492 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:06.828091 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:06.828116 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:06.830027 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:06.832822 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:06.832830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:06.832846 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:06.833409 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:06.833834 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:06.833839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:06.833842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:06.834111 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:06.834619 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:06.834626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:06.834948 (+ 322us) client_negotiation.cc:769] Sending connection context
0926 19:42:06.835198 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 19:42:06.835718 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6901,"thread_start_us":6662,"threads_started":1}
I20260926 19:42:06.838116 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:06.811107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:42697 (local address 127.2.164.126:40225)
0926 19:42:06.817071 (+ 5964us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:06.817078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:06.818207 (+ 1129us) server_negotiation.cc:407] Connection header received
0926 19:42:06.820667 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:06.820674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:06.820768 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:06.820862 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:06.822848 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:06.823740 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:06.826214 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:06.826923 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:06.827022 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:06.830232 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:06.830237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:06.830240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:06.830271 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:06.832700 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:06.833552 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:06.833558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:06.833561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:06.833708 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:06.834225 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:06.834229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:06.834231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:06.834504 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:06.834631 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:06.837540 (+ 2909us) server_negotiation.cc:299] Negotiation successful
0926 19:42:06.837760 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5836,"thread_start_us":5717,"threads_started":1}
I20260926 19:42:06.840101 15885 heartbeater.cc:347] Connected to a master server at 127.2.164.126:40225
I20260926 19:42:06.840387 15885 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:06.841049 15885 heartbeater.cc:511] Master 127.2.164.126:40225 requested a full tablet report, sending...
I20260926 19:42:06.842715 15543 ts_manager.cc:194] Registered new tserver with Master: b34d470b95bf483b9eb985f0de126961 (127.2.164.66:40187)
I20260926 19:42:06.844220 15543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:42697
Sep 26 19:42:07 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451726, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:42:07.188421 15740 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:07.188841 15740 heartbeater.cc:511] Master 127.2.164.126:40225 requested a full tablet report, sending...
I20260926 19:42:07.190184 15543 ts_manager.cc:194] Registered new tserver with Master: 2d71a77f3c71479ba91cb9578e5eb4e1 (127.2.164.65:44779)
I20260926 19:42:07.191308 15543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:55403
W20260926 19:42:07.242127 15889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:07.242408 15889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:07.242539 15889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:07.249841 15889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:07.249979 15889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:07.250185 15889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:42:07.265151 15889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.164.84:46971/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:34327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=33557
--tserver_master_addrs=127.2.164.126:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:07.266779 15889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:07.268386 15889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:07.284158 15894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:07.285521 15895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:07.289793 15889 server_base.cc:1034] running on GCE node
W20260926 19:42:07.290064 15897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:07.290849 15889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:07.292058 15889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:07.293304 15889 hybrid_clock.cc:648] HybridClock initialized: now 1790451727293233 us; error 57 us; skew 500 ppm
I20260926 19:42:07.293671 15889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:07 dist-test-slave-mrw4 krb5kdc[14533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451727, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:07.297600 15889 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:42:07.299650 15889 webserver.cc:460] Webserver started at http://127.2.164.67:33557/ using document root <none> and password file <none>
I20260926 19:42:07.300505 15889 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:07.300644 15889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 19:42:07 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451727, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:42:07.331303 15889 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 19:42:07.339392 15905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:07.341487 15889 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.008s sys 0.000s
I20260926 19:42:07.341727 15889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "acc2ec14cf0d45b98f3b15f8f8525f83"
format_stamp: "Formatted at 2026-09-26 19:42:04 on dist-test-slave-mrw4"
server_key: "1e7f8551880d04eb537f2baeb2fb7162"
server_key_iv: "202010794d3e4684175a71826c4078a7"
server_key_version: "kuduclusterkey@0"
I20260926 19:42:07.343070 15889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:07.364599 15889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:07.369751 15889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:07.370093 15889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:07.376509 15889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:07.378820 15889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:07.378964 15889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 19:42:07.379184 15889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:07.379300 15889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:42:07 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451726, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for HTTP/127.2.164.84@KRBTEST.COM
I20260926 19:42:07.578830 15889 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:34327
I20260926 19:42:07.578871 16018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:34327 every 8 connection(s)
I20260926 19:42:07.580744 15889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:42:07.582528 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 15889
I20260926 19:42:07.613559 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.584967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.2.164.126:40225)
0926 19:42:07.586101 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:07.586109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:07.590445 (+ 4336us) server_negotiation.cc:407] Connection header received
0926 19:42:07.590535 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:07.590540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:07.590623 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:07.590723 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:07.593819 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.594958 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.599183 (+ 4225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.599844 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.599946 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.603649 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:07.603657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:07.603661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:07.603704 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:07.607356 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:07.608220 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:07.608226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:07.608239 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:07.608391 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:07.608859 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:07.608864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:07.608867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:07.609285 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:07.609561 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:07.612912 (+ 3351us) server_negotiation.cc:299] Negotiation successful
0926 19:42:07.613129 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":843,"threads_started":1}
Sep 26 19:42:07 dist-test-slave-mrw4 krb5kdc[14533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451727, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:07.630528 16024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.585351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:58453 (local address 127.2.164.126:40225)
0926 19:42:07.609619 (+ 24268us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:07.609625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:07.609645 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:42:07.609712 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:07.609718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:07.609790 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:07.609900 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:07.612161 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.613413 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.615924 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.616597 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.616723 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.620897 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:07.620904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:07.620909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:07.620946 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:07.624314 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:07.625694 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:07.625700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:07.625703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:07.625840 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:07.626408 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:07.626412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:07.626416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:07.626800 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:07.626922 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:07.629907 (+ 2985us) server_negotiation.cc:299] Negotiation successful
0926 19:42:07.630129 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24123,"thread_start_us":24691,"threads_started":1}
I20260926 19:42:07.631405 16022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.584291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:40225 (local address 127.2.164.67:58453)
0926 19:42:07.587076 (+ 2785us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:07.587133 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:07.596877 (+ 9744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:07.610293 (+ 13416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:07.610309 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:07.610985 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:07.611879 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.611907 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.613606 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.613612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.615711 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.615727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.616834 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.616842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.617143 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.617916 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:07.617947 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:07.620610 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:07.624852 (+ 4242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:07.624861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:07.624879 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:07.625556 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:07.625974 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:07.625978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:07.625981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:07.626270 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:07.627003 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:07.627010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:07.629056 (+ 2046us) client_negotiation.cc:769] Sending connection context
0926 19:42:07.629415 (+ 359us) client_negotiation.cc:240] Negotiation successful
0926 19:42:07.629978 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2459,"thread_start_us":2245,"threads_started":1}
I20260926 19:42:07.633221 16019 heartbeater.cc:347] Connected to a master server at 127.2.164.126:40225
I20260926 19:42:07.633539 16019 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:07.634351 16019 heartbeater.cc:511] Master 127.2.164.126:40225 requested a full tablet report, sending...
I20260926 19:42:07.635949 15543 ts_manager.cc:194] Registered new tserver with Master: acc2ec14cf0d45b98f3b15f8f8525f83 (127.2.164.67:34327)
I20260926 19:42:07.637001 15543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:58453
I20260926 19:42:07.641021 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:42:07.661479 16024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.644367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.2.164.126:40225)
0926 19:42:07.644536 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:07.644541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:07.645705 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 19:42:07.645866 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:07.645872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:07.645948 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:07.646047 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:07.647831 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.649005 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.650897 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.651568 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.651694 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.654857 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:07.654863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:07.654869 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:07.654907 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:07.658306 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:07.659266 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:07.659272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:07.659275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:07.659424 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:07.659869 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:07.659874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:07.659877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:07.660309 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:07.660545 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:07.660900 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 19:42:07.661121 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:42:07.667091 15543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:42:07.678668 15543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:42:07.728821 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.697040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:34327 (local address 127.0.0.1:36668)
0926 19:42:07.713265 (+ 16225us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:07.713303 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:07.713497 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:07.720373 (+ 6876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:07.720382 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:07.720414 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:07.720781 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.720794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.722681 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.722687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.724335 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.724356 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.727712 (+ 3356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.727719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.727937 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.727989 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 19:42:07.728136 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 19:42:07.728356 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15937,"mutex_wait_us":3,"thread_start_us":15806,"threads_started":1}
I20260926 19:42:07.729393 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.698111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.2.164.66:40187)
0926 19:42:07.720699 (+ 22588us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:07.720707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:07.720734 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:42:07.720811 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:07.720816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:07.721033 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:07.721685 (+ 652us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:07.722521 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.724163 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.726375 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.727727 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.727861 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.728043 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:07.728927 (+ 884us) server_negotiation.cc:299] Negotiation successful
0926 19:42:07.729086 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22464,"thread_start_us":22350,"threads_started":1}
I20260926 19:42:07.730729 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.697540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.2.164.67:34327)
0926 19:42:07.719381 (+ 21841us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:07.719389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:07.719422 (+ 33us) server_negotiation.cc:407] Connection header received
0926 19:42:07.719527 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:07.719536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:07.719844 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:07.720044 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:07.720987 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.722561 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.724498 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.727184 (+ 2686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.727308 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.727539 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:07.730233 (+ 2694us) server_negotiation.cc:299] Negotiation successful
0926 19:42:07.730402 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21677,"thread_start_us":21530,"threads_started":1}
I20260926 19:42:07.729857 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.697244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:40187 (local address 127.0.0.1:56242)
0926 19:42:07.718527 (+ 21283us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:07.718548 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:07.718673 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:07.721912 (+ 3239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:07.721917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:07.721945 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:07.722364 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.722374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.724295 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.724304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.726203 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.726217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.728213 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.728219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.728387 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.728420 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:42:07.728512 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 19:42:07.728631 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21157,"mutex_wait_us":21,"thread_start_us":23943,"threads_started":1}
I20260926 19:42:07.731992 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.698538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.2.164.65:44779)
0926 19:42:07.718781 (+ 20243us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:07.718790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:07.721239 (+ 2449us) server_negotiation.cc:407] Connection header received
0926 19:42:07.721325 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:07.721331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:07.721487 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:07.721646 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:07.725722 (+ 4076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.727226 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.729595 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.730710 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.730792 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.730996 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:07.731561 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 19:42:07.731709 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20068,"thread_start_us":20255,"threads_started":1}
I20260926 19:42:07.732643 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.698347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44779 (local address 127.0.0.1:47130)
0926 19:42:07.720125 (+ 21778us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:07.720144 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:07.720267 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:07.721870 (+ 1603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:07.721874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:07.721895 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:07.725443 (+ 3548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.725458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.727447 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.727452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.729428 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.729443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.731196 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.731203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.731376 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.731407 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:42:07.732168 (+ 761us) client_negotiation.cc:240] Negotiation successful
0926 19:42:07.732293 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21656,"mutex_wait_us":3074,"thread_start_us":20422,"threads_started":1}
I20260926 19:42:07.734930 15954 tablet_service.cc:1467] Processing CreateTablet for tablet d6ad7b9c62524969a159ca6e5ad4296b (DEFAULT_TABLE table=test-table [id=30627b9915974250aa977c8a98914fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:07.734932 15820 tablet_service.cc:1467] Processing CreateTablet for tablet d6ad7b9c62524969a159ca6e5ad4296b (DEFAULT_TABLE table=test-table [id=30627b9915974250aa977c8a98914fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:07.736272 15954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ad7b9c62524969a159ca6e5ad4296b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:07.736277 15820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ad7b9c62524969a159ca6e5ad4296b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:07.735877 15675 tablet_service.cc:1467] Processing CreateTablet for tablet d6ad7b9c62524969a159ca6e5ad4296b (DEFAULT_TABLE table=test-table [id=30627b9915974250aa977c8a98914fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:07.737015 15675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ad7b9c62524969a159ca6e5ad4296b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:07.756134 16048 tablet_bootstrap.cc:492] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: Bootstrap starting.
I20260926 19:42:07.758579 16047 tablet_bootstrap.cc:492] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1: Bootstrap starting.
I20260926 19:42:07.763247 16048 tablet_bootstrap.cc:654] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:07.763657 16047 tablet_bootstrap.cc:654] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:07.764379 16049 tablet_bootstrap.cc:492] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: Bootstrap starting.
I20260926 19:42:07.764744 16048 log.cc:824] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:07.767295 16049 tablet_bootstrap.cc:654] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:07.767946 16048 tablet_bootstrap.cc:492] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: No bootstrap required, opened a new log
I20260926 19:42:07.768225 16048 ts_tablet_manager.cc:1397] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20260926 19:42:07.768250 16049 log.cc:824] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:07.769788 16047 log.cc:824] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:07.772141 16049 tablet_bootstrap.cc:492] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: No bootstrap required, opened a new log
I20260926 19:42:07.772374 16049 ts_tablet_manager.cc:1397] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 19:42:07.778375 16048 raft_consensus.cc:348] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34327 } } peers { permanent_uuid: "b34d470b95bf483b9eb985f0de126961" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40187 } } peers { permanent_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44779 } }
I20260926 19:42:07.779469 16049 raft_consensus.cc:348] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34327 } } peers { permanent_uuid: "b34d470b95bf483b9eb985f0de126961" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40187 } } peers { permanent_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44779 } }
I20260926 19:42:07.779724 16048 raft_consensus.cc:374] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:07.779878 16049 raft_consensus.cc:374] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:07.779884 16048 raft_consensus.cc:729] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc2ec14cf0d45b98f3b15f8f8525f83, State: Initialized, Role: FOLLOWER
I20260926 19:42:07.780055 16049 raft_consensus.cc:729] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34d470b95bf483b9eb985f0de126961, State: Initialized, Role: FOLLOWER
I20260926 19:42:07.780483 16048 consensus_queue.cc:260] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34327 } } peers { permanent_uuid: "b34d470b95bf483b9eb985f0de126961" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40187 } } peers { permanent_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44779 } }
I20260926 19:42:07.780702 16049 consensus_queue.cc:260] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34327 } } peers { permanent_uuid: "b34d470b95bf483b9eb985f0de126961" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40187 } } peers { permanent_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44779 } }
I20260926 19:42:07.781694 16047 tablet_bootstrap.cc:492] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1: No bootstrap required, opened a new log
I20260926 19:42:07.783527 16047 ts_tablet_manager.cc:1397] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.000s
I20260926 19:42:07.793838 15885 heartbeater.cc:503] Master 127.2.164.126:40225 was elected leader, sending a full tablet report...
I20260926 19:42:07.794548 16049 ts_tablet_manager.cc:1428] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: Time spent starting tablet: real 0.022s user 0.014s sys 0.008s
I20260926 19:42:07.795027 16019 heartbeater.cc:503] Master 127.2.164.126:40225 was elected leader, sending a full tablet report...
I20260926 19:42:07.794777 16047 raft_consensus.cc:348] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34327 } } peers { permanent_uuid: "b34d470b95bf483b9eb985f0de126961" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40187 } } peers { permanent_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44779 } }
I20260926 19:42:07.795218 16047 raft_consensus.cc:374] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:07.795301 16047 raft_consensus.cc:729] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d71a77f3c71479ba91cb9578e5eb4e1, State: Initialized, Role: FOLLOWER
I20260926 19:42:07.795974 16047 consensus_queue.cc:260] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34327 } } peers { permanent_uuid: "b34d470b95bf483b9eb985f0de126961" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40187 } } peers { permanent_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44779 } }
I20260926 19:42:07.796921 16048 ts_tablet_manager.cc:1428] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: Time spent starting tablet: real 0.029s user 0.017s sys 0.009s
I20260926 19:42:07.798820 15740 heartbeater.cc:503] Master 127.2.164.126:40225 was elected leader, sending a full tablet report...
I20260926 19:42:07.799999 16047 ts_tablet_manager.cc:1428] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1: Time spent starting tablet: real 0.016s user 0.012s sys 0.001s
W20260926 19:42:07.822148 15886 tablet.cc:2369] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:42:07.846132 16020 tablet.cc:2369] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:42:07.909118 15741 tablet.cc:2369] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:42:07.966367 16053 raft_consensus.cc:482] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:42:07.966645 16053 raft_consensus.cc:504] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34327 } } peers { permanent_uuid: "b34d470b95bf483b9eb985f0de126961" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40187 } } peers { permanent_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44779 } }
I20260926 19:42:07.968330 16053 leader_election.cc:288] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b34d470b95bf483b9eb985f0de126961 (127.2.164.66:40187), 2d71a77f3c71479ba91cb9578e5eb4e1 (127.2.164.65:44779)
I20260926 19:42:07.975494 16022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.968635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:40187 (local address 127.2.164.67:39039)
0926 19:42:07.968816 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:07.968841 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:07.968965 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:07.970070 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:07.970075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:07.970104 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:07.970368 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.970381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.971720 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.971725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.973739 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.973750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.974808 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.974814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.974974 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.974998 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 19:42:07.975084 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 19:42:07.975224 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":45}
I20260926 19:42:07.975651 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.969505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:39039 (local address 127.2.164.66:40187)
0926 19:42:07.969706 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:07.969710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:07.969724 (+ 14us) server_negotiation.cc:407] Connection header received
0926 19:42:07.969787 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:07.969791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:07.969849 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:07.969929 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:07.970544 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.971562 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.973889 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.974654 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.974802 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.974886 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:07.975269 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 19:42:07.975404 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:42:07.975826 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.969232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:43755 (local address 127.2.164.65:44779)
0926 19:42:07.969424 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:07.969430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:07.969548 (+ 118us) server_negotiation.cc:407] Connection header received
0926 19:42:07.969713 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:07.969718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:07.969794 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:07.969884 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:07.970468 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.971235 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.973774 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.974952 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.975108 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.975177 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:07.975482 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 19:42:07.975616 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:42:07.976438 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:07.968721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44779 (local address 127.2.164.67:43755)
0926 19:42:07.969458 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:07.969474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:07.969586 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:07.969915 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:07.969920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:07.969945 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:07.970300 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.970314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.971370 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.971374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.973596 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:07.973611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:07.975135 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:07.975141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:07.975305 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:07.975338 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:42:07.975984 (+ 646us) client_negotiation.cc:240] Negotiation successful
0926 19:42:07.976110 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":1,"thread_start_us":424,"threads_started":1}
I20260926 19:42:07.977021 15840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ad7b9c62524969a159ca6e5ad4296b" candidate_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b34d470b95bf483b9eb985f0de126961" is_pre_election: true
I20260926 19:42:07.977378 15695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ad7b9c62524969a159ca6e5ad4296b" candidate_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" is_pre_election: true
I20260926 19:42:07.977592 15840 raft_consensus.cc:2454] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc2ec14cf0d45b98f3b15f8f8525f83 in term 0.
I20260926 19:42:07.977856 15695 raft_consensus.cc:2454] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc2ec14cf0d45b98f3b15f8f8525f83 in term 0.
I20260926 19:42:07.978299 15907 leader_election.cc:302] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc2ec14cf0d45b98f3b15f8f8525f83, b34d470b95bf483b9eb985f0de126961; no voters:
I20260926 19:42:07.978690 16053 raft_consensus.cc:2781] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:42:07.978820 16053 raft_consensus.cc:482] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:42:07.978927 16053 raft_consensus.cc:3037] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:07.980938 16053 raft_consensus.cc:504] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34327 } } peers { permanent_uuid: "b34d470b95bf483b9eb985f0de126961" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40187 } } peers { permanent_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44779 } }
I20260926 19:42:07.981515 16053 leader_election.cc:288] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [CANDIDATE]: Term 1 election: Requested vote from peers b34d470b95bf483b9eb985f0de126961 (127.2.164.66:40187), 2d71a77f3c71479ba91cb9578e5eb4e1 (127.2.164.65:44779)
I20260926 19:42:07.981897 15840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ad7b9c62524969a159ca6e5ad4296b" candidate_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b34d470b95bf483b9eb985f0de126961"
I20260926 19:42:07.981984 15695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ad7b9c62524969a159ca6e5ad4296b" candidate_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1"
I20260926 19:42:07.982092 15840 raft_consensus.cc:3037] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:07.982163 15695 raft_consensus.cc:3037] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:07.985083 15840 raft_consensus.cc:2454] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc2ec14cf0d45b98f3b15f8f8525f83 in term 1.
I20260926 19:42:07.985085 15695 raft_consensus.cc:2454] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc2ec14cf0d45b98f3b15f8f8525f83 in term 1.
I20260926 19:42:07.985484 15908 leader_election.cc:302] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d71a77f3c71479ba91cb9578e5eb4e1, acc2ec14cf0d45b98f3b15f8f8525f83; no voters:
I20260926 19:42:07.985731 16053 raft_consensus.cc:2781] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:42:07.986155 16053 raft_consensus.cc:686] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 1 LEADER]: Becoming Leader. State: Replica: acc2ec14cf0d45b98f3b15f8f8525f83, State: Running, Role: LEADER
I20260926 19:42:07.986601 16053 consensus_queue.cc:237] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34327 } } peers { permanent_uuid: "b34d470b95bf483b9eb985f0de126961" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40187 } } peers { permanent_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44779 } }
I20260926 19:42:07.991245 15542 catalog_manager.cc:5495] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc2ec14cf0d45b98f3b15f8f8525f83 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc2ec14cf0d45b98f3b15f8f8525f83" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b34d470b95bf483b9eb985f0de126961" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44779 } health_report { overall_health: UNKNOWN } } }
I20260926 19:42:08.024484 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.019257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.2.164.67:34327)
0926 19:42:08.019423 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:08.019428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:08.019508 (+ 80us) server_negotiation.cc:407] Connection header received
0926 19:42:08.019708 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:08.019712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:08.019797 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:08.019915 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:42:08.020238 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:08.020986 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:08.022490 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:08.022871 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:08.022978 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:08.023449 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:42:08.023935 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:42:08.024069 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:08.024156 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 19:42:08.024234 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 19:42:08.031960 15840 raft_consensus.cc:1261] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961 [term 1 FOLLOWER]: Refusing update from remote peer acc2ec14cf0d45b98f3b15f8f8525f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:42:08.031955 15695 raft_consensus.cc:1261] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1 [term 1 FOLLOWER]: Refusing update from remote peer acc2ec14cf0d45b98f3b15f8f8525f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:42:08.032835 16057 consensus_queue.cc:1035] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b34d470b95bf483b9eb985f0de126961" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 40187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:42:08.033120 16053 consensus_queue.cc:1035] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d71a77f3c71479ba91cb9578e5eb4e1" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:42:08.045461 16060 mvcc.cc:205] Tried to move back new op lower bound from 7333690278010372096 to 7333690277836816384. Current Snapshot: MvccSnapshot[applied={T|T < 7333690278010372096}]
I20260926 19:42:08.059511 15543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:42:08.059787 15543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:42:08.063242 15543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa193ca8ee4c4ba3b75c1ecd40476911: Sending DeleteTablet for 3 replicas of tablet d6ad7b9c62524969a159ca6e5ad4296b
I20260926 19:42:08.064307 15954 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ad7b9c62524969a159ca6e5ad4296b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:08 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:36668
I20260926 19:42:08.064441 15675 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ad7b9c62524969a159ca6e5ad4296b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:08 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:47130
I20260926 19:42:08.064543 15820 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ad7b9c62524969a159ca6e5ad4296b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:08 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:56242
I20260926 19:42:08.065240 16070 tablet_replica.cc:330] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: stopping tablet replica
I20260926 19:42:08.065692 16070 raft_consensus.cc:2229] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:42:08.066351 16070 raft_consensus.cc:2258] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:42:08.067610 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 15580
I20260926 19:42:08.076748 16070 ts_tablet_manager.cc:1905] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:42:08.078369 16072 tablet_replica.cc:330] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: stopping tablet replica
I20260926 19:42:08.078796 16072 raft_consensus.cc:2229] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:42:08.079268 16072 raft_consensus.cc:2258] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:42:08.080039 16071 tablet_replica.cc:330] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1: stopping tablet replica
I20260926 19:42:08.080456 16071 raft_consensus.cc:2229] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:42:08.080967 16071 raft_consensus.cc:2258] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:42:08.081895 16072 ts_tablet_manager.cc:1905] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:42:08.084880 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.068764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.2.164.65:44779)
0926 19:42:08.069322 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:08.069327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:08.069344 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:42:08.069477 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:08.069482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:08.069556 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:08.069638 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:08.071517 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:08.072743 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:08.074041 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:08.074729 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:08.074847 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:08.077897 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:08.077915 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:08.077935 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:08.077978 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:08.082217 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:08.082857 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:08.082865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:08.082868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:08.083015 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:08.083363 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:08.083368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:08.083371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:08.083771 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:08.084055 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:08.084345 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 19:42:08.084539 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426}
I20260926 19:42:08.085716 15735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:08.087672 16070 ts_tablet_manager.cc:1918] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:42:08.087865 16070 log.cc:1196] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/d6ad7b9c62524969a159ca6e5ad4296b
I20260926 19:42:08.088451 16070 ts_tablet_manager.cc:1939] T d6ad7b9c62524969a159ca6e5ad4296b P acc2ec14cf0d45b98f3b15f8f8525f83: Deleting consensus metadata
I20260926 19:42:08.090273 15527 catalog_manager.cc:4841] TS acc2ec14cf0d45b98f3b15f8f8525f83 (127.2.164.67:34327): tablet d6ad7b9c62524969a159ca6e5ad4296b (table test-table [id=30627b9915974250aa977c8a98914fc0]) successfully deleted
I20260926 19:42:08.090639 16072 ts_tablet_manager.cc:1918] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:42:08.090823 16072 log.cc:1196] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/d6ad7b9c62524969a159ca6e5ad4296b
I20260926 19:42:08.091352 16072 ts_tablet_manager.cc:1939] T d6ad7b9c62524969a159ca6e5ad4296b P b34d470b95bf483b9eb985f0de126961: Deleting consensus metadata
I20260926 19:42:08.094082 15528 catalog_manager.cc:4841] TS b34d470b95bf483b9eb985f0de126961 (127.2.164.66:40187): tablet d6ad7b9c62524969a159ca6e5ad4296b (table test-table [id=30627b9915974250aa977c8a98914fc0]) successfully deleted
I20260926 19:42:08.179410 16071 ts_tablet_manager.cc:1905] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:42:08.184350 16071 ts_tablet_manager.cc:1918] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:42:08.184490 16071 log.cc:1196] T d6ad7b9c62524969a159ca6e5ad4296b P 2d71a77f3c71479ba91cb9578e5eb4e1: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/d6ad7b9c62524969a159ca6e5ad4296b
I20260926 19:42:08.184800 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 15580
W20260926 19:42:08.198386 15529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:44779 (error 108)
I20260926 19:42:08.199586 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 15744
I20260926 19:42:08.200201 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.199619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44779 (local address 127.0.0.1:47148)
0926 19:42:08.199901 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:08.200025 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260926 19:42:08.200568 15529 catalog_manager.cc:4584] TS 2d71a77f3c71479ba91cb9578e5eb4e1 (127.2.164.65:44779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6ad7b9c62524969a159ca6e5ad4296b: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44779: connect: Connection refused (error 111)
I20260926 19:42:08.216468 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.200285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.2.164.66:40187)
0926 19:42:08.200452 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:08.200458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:08.200551 (+ 93us) server_negotiation.cc:407] Connection header received
0926 19:42:08.200721 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:08.200726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:08.200829 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:08.200945 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:08.202266 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:08.202963 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:08.204227 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:08.204612 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:08.204719 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:08.210919 (+ 6200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:08.210935 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:08.210951 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:08.210985 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:08.213722 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:08.214387 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:08.214415 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:08.214417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:08.214552 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:08.214888 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:08.214893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:08.214896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:08.215299 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:08.215599 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:08.215896 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 19:42:08.216150 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:42:08.217195 15880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:08.254258 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.253750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44779 (local address 127.0.0.1:47156)
0926 19:42:08.253970 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:08.254090 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 19:42:08.255396 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.254992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44779 (local address 127.0.0.1:47166)
0926 19:42:08.255148 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:08.255249 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 19:42:08.304855 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 15744
I20260926 19:42:08.315780 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.315337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44779 (local address 127.0.0.1:47180)
0926 19:42:08.315516 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:08.315610 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 19:42:08.316844 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.316496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44779 (local address 127.0.0.1:47192)
0926 19:42:08.316634 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:08.316709 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 19:42:08.324621 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 15889
I20260926 19:42:08.337775 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.325315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.2.164.67:34327)
0926 19:42:08.325517 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:08.325523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:08.325544 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:42:08.325718 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:08.325726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:08.325788 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:08.325866 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:08.327149 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:08.327912 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:08.329221 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:08.329673 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:08.329789 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:08.332108 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:08.332120 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:08.332138 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:08.332178 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:08.335072 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:08.335713 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:08.335720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:08.335723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:08.335876 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:08.336206 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:08.336210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:08.336212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:08.336557 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:08.336852 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:08.337228 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 19:42:08.337438 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:42:08.338402 16014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:08.397509 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.397002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44779 (local address 127.0.0.1:47198)
0926 19:42:08.397245 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:08.397332 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 19:42:08.398561 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.398192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44779 (local address 127.0.0.1:47212)
0926 19:42:08.398380 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:08.398454 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 19:42:08.457511 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 15889
I20260926 19:42:08.482034 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 15510
I20260926 19:42:08.483103 15573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:08.591619 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 15510
I20260926 19:42:08.591715 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:08.591036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44779 (local address 127.0.0.1:47226)
0926 19:42:08.591391 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:08.591495 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:44779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260926 19:42:08.611711 2705 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 19:42:08.945773 2705 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 19:42:08.945968 2705 mini_ranger.cc:66] Stopping Ranger...
I20260926 19:42:09.280613 2705 mini_ranger.cc:68] Stopped Ranger
2026-09-26 19:42:09.280 UTC [14563] LOG: received smart shutdown request
2026-09-26 19:42:09.283 UTC [14563] LOG: background worker "logical replication launcher" (PID 14572) exited with exit code 1
2026-09-26 19:42:09.284 UTC [14567] LOG: shutting down
2026-09-26 19:42:09.441 UTC [14563] LOG: database system is shut down
2026-09-26T19:42:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50024 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:42:09 dist-test-slave-mrw4 krb5kdc[16086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:42:09 dist-test-slave-mrw4 krb5kdc[16086](info): set up 2 sockets
Sep 26 19:42:09 dist-test-slave-mrw4 krb5kdc[16086](info): commencing operation
krb5kdc: starting...
W20260926 19:42:10.549321 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:42:10 dist-test-slave-mrw4 krb5kdc[16086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:42:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:42:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:42:10.750855 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33707
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:33707,127.2.164.125:35187,127.2.164.124:42795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:11.055464 16099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:11.055754 16099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:11.055891 16099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:11.063042 16099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:42:11.063164 16099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:11.063242 16099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:11.063313 16099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:42:11.063370 16099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:42:11.073127 16099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:33707,127.2.164.125:35187,127.2.164.124:42795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:33707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:11.074445 16099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:11.075704 16099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:11.086803 16104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:11.086874 16105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:11.086823 16107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:11.087491 16099 server_base.cc:1034] running on GCE node
I20260926 19:42:11.088196 16099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:11.089524 16099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:11.090723 16099 hybrid_clock.cc:648] HybridClock initialized: now 1790451731090674 us; error 41 us; skew 500 ppm
I20260926 19:42:11.091033 16099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:11 dist-test-slave-mrw4 krb5kdc[16086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:11.093935 16099 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:42:11.095355 16099 webserver.cc:460] Webserver started at http://127.2.164.126:33207/ using document root <none> and password file <none>
I20260926 19:42:11.095988 16099 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:11.096098 16099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:11.096362 16099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:11.099165 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "64c8012f6f394a9fa703b95465058d7d"
format_stamp: "Formatted at 2026-09-26 19:42:11 on dist-test-slave-mrw4"
server_key: "b5fcb12b7206c607143d850d6e2f3ed9"
server_key_iv: "4da24cd6075fb073b232485f8f0abbc6"
server_key_version: "encryptionkey@0"
I20260926 19:42:11.099918 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "64c8012f6f394a9fa703b95465058d7d"
format_stamp: "Formatted at 2026-09-26 19:42:11 on dist-test-slave-mrw4"
server_key: "b5fcb12b7206c607143d850d6e2f3ed9"
server_key_iv: "4da24cd6075fb073b232485f8f0abbc6"
server_key_version: "encryptionkey@0"
I20260926 19:42:11.104593 16099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 19:42:11.107762 16114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:11.108981 16099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 19:42:11.109134 16099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "64c8012f6f394a9fa703b95465058d7d"
format_stamp: "Formatted at 2026-09-26 19:42:11 on dist-test-slave-mrw4"
server_key: "b5fcb12b7206c607143d850d6e2f3ed9"
server_key_iv: "4da24cd6075fb073b232485f8f0abbc6"
server_key_version: "encryptionkey@0"
I20260926 19:42:11.109326 16099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:11.121768 16099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:11.125473 16099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:11.125702 16099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:11.142386 16099 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:33707
I20260926 19:42:11.142418 16165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:33707 every 8 connection(s)
I20260926 19:42:11.143632 16099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:42:11.147136 16166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:11.148628 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 16099
I20260926 19:42:11.148823 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:42:11.149473 2705 external_mini_cluster.cc:1428] Setting key 9fd69b01582cec2d3e17af27440514f3
I20260926 19:42:11.153510 16166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.164.125@KRBTEST.COM; defaulting to no policy
Sep 26 19:42:11 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.125@KRBTEST.COM" created.
I20260926 19:42:11.176503 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.156626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33707 (local address 127.0.0.1:47948)
0926 19:42:11.157511 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.157564 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:11.159851 (+ 2287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:11.160693 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:11.160699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:11.161409 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:11.161825 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:11.161848 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.163314 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.163319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:11.164940 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:11.164957 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.165913 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.165920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:11.166086 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:11.166714 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:11.166743 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:11.169569 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:11.172502 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:11.172513 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:11.172524 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:11.172914 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:11.173321 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:11.173326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:11.173330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:11.173473 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:11.174180 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:11.174187 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:11.174389 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 19:42:11.174595 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 19:42:11.175182 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":129,"thread_start_us":365,"threads_started":1}
I20260926 19:42:11.176476 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.156626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.2.164.126:33707)
0926 19:42:11.158379 (+ 1753us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:11.158385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:11.158430 (+ 45us) server_negotiation.cc:407] Connection header received
0926 19:42:11.160236 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:11.160242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:11.160471 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:11.160616 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:11.162010 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.163156 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.165072 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.165668 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.165790 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:11.169767 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:11.169774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:11.169785 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:11.169814 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:11.172384 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:11.173039 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:11.173046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:11.173048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:11.173215 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:11.173577 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:11.173582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:11.173586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:11.174052 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:11.174264 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:11.174750 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 19:42:11.175026 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1571,"spinlock_wait_cycles":71936,"thread_start_us":1346,"threads_started":1}
I20260926 19:42:11.179733 16166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:42:11.180822 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.180398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51646)
0926 19:42:11.180571 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.180688 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 19:42:11.181070 16118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
I20260926 19:42:11.182574 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.182188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51650)
0926 19:42:11.182333 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.182416 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 19:42:11.182780 16166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:35187: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab.
Entry for principal kudu/127.2.164.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab.
I20260926 19:42:11.195749 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.125:35187
--webserver_interface=127.2.164.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab
--principal=kudu/127.2.164.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:33707,127.2.164.125:35187,127.2.164.124:42795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:42:11.232079 16166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } attempt: 1
I20260926 19:42:11.233268 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.232782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51654)
0926 19:42:11.232978 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.233106 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 19:42:11.234225 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.233919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51664)
0926 19:42:11.234042 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.234108 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 19:42:11.234421 16166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:35187: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
I20260926 19:42:11.302652 16166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } attempt: 2
I20260926 19:42:11.303835 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.303366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51676)
0926 19:42:11.303596 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.303687 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 19:42:11.304849 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.304472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51692)
0926 19:42:11.304628 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.304698 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 19:42:11.305032 16166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:35187: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
I20260926 19:42:11.396275 16166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } attempt: 3
I20260926 19:42:11.397454 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.396994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51704)
0926 19:42:11.397210 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.397324 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 19:42:11.398433 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.398087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51714)
0926 19:42:11.398228 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.398303 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 19:42:11.398615 16166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:35187: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
W20260926 19:42:11.500694 16173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:11.500965 16173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:11.501092 16173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:11.508113 16173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:42:11.508256 16173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:11.508337 16173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:11.508407 16173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:42:11.508467 16173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:42:11.518289 16173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:33707,127.2.164.125:35187,127.2.164.124:42795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab
--principal=kudu/127.2.164.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.125:35187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.164.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:11.519570 16173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:11.520833 16173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:11.531543 16179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:11.531622 16178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:11.531635 16181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:11.532656 16173 server_base.cc:1034] running on GCE node
I20260926 19:42:11.533435 16173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:11.534699 16173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:11.535893 16173 hybrid_clock.cc:648] HybridClock initialized: now 1790451731535844 us; error 36 us; skew 500 ppm
I20260926 19:42:11.536201 16173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:42:11.541827 16166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } attempt: 4
I20260926 19:42:11.542932 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.542331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51728)
0926 19:42:11.542651 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.542751 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
Sep 26 19:42:11 dist-test-slave-mrw4 krb5kdc[16086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:11.543831 16173 init.cc:377] Logged in from keytab as kudu/127.2.164.125@KRBTEST.COM (short username kudu)
I20260926 19:42:11.543948 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.543537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51734)
0926 19:42:11.543684 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.543788 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 19:42:11.544176 16166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:35187: Network error: Client connection negotiation failed: client connection to 127.2.164.125:35187: connect: Connection refused (error 111)
I20260926 19:42:11.545341 16173 webserver.cc:460] Webserver started at http://127.2.164.125:33339/ using document root <none> and password file <none>
I20260926 19:42:11.545954 16173 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:11.546047 16173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:11.546283 16173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:11.549026 16173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/data/instance:
uuid: "53315d8e48de404e87330859a61884a6"
format_stamp: "Formatted at 2026-09-26 19:42:11 on dist-test-slave-mrw4"
server_key: "c2f29cc430cd09cfca0413dc5bba7fa7"
server_key_iv: "b58ad793277befefaa45b6ca85289ac5"
server_key_version: "encryptionkey@0"
I20260926 19:42:11.549794 16173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/wal/instance:
uuid: "53315d8e48de404e87330859a61884a6"
format_stamp: "Formatted at 2026-09-26 19:42:11 on dist-test-slave-mrw4"
server_key: "c2f29cc430cd09cfca0413dc5bba7fa7"
server_key_iv: "b58ad793277befefaa45b6ca85289ac5"
server_key_version: "encryptionkey@0"
I20260926 19:42:11.554384 16173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:42:11.557435 16188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:11.558660 16173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:42:11.558787 16173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/wal
uuid: "53315d8e48de404e87330859a61884a6"
format_stamp: "Formatted at 2026-09-26 19:42:11 on dist-test-slave-mrw4"
server_key: "c2f29cc430cd09cfca0413dc5bba7fa7"
server_key_iv: "b58ad793277befefaa45b6ca85289ac5"
server_key_version: "encryptionkey@0"
I20260926 19:42:11.558930 16173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:11.583765 16173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:11.587484 16173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:11.587698 16173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:11.603958 16173 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.125:35187
I20260926 19:42:11.603991 16239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.125:35187 every 8 connection(s)
I20260926 19:42:11.605212 16173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
I20260926 19:42:11.608758 16240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:11.613688 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 16173
I20260926 19:42:11.613866 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-1/wal/instance
I20260926 19:42:11.614285 2705 external_mini_cluster.cc:1428] Setting key e8d8b6ee1ae723e5e02e39f67190558d
I20260926 19:42:11.615005 16240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:42:11 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.124@KRBTEST.COM; defaulting to no policy
I20260926 19:42:11.634617 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.617394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.2.164.126:33707)
0926 19:42:11.617555 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:11.617560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:11.618914 (+ 1354us) server_negotiation.cc:407] Connection header received
0926 19:42:11.620696 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:11.620701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:11.620773 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:11.620849 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:11.622525 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.623476 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.624937 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.625568 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.625685 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:11.629067 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:11.629072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:11.629075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:11.629108 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:11.631550 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:11.632432 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:11.632437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:11.632439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:11.632581 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:11.632987 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:11.632991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:11.632993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:11.633341 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:11.633501 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:11.634141 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 19:42:11.634316 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.124@KRBTEST.COM" created.
I20260926 19:42:11.635468 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.618052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33707 (local address 127.0.0.1:47956)
0926 19:42:11.618761 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.618801 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:11.620397 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:11.621094 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:11.621105 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:11.621822 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:11.622356 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:11.622372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.623625 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.623629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:11.624812 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:11.624824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.625758 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.625763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:11.625946 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:11.626575 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:11.626597 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:11.628834 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:11.631694 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:11.631702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:11.631719 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:11.632299 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:11.632682 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:11.632686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:11.632689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:11.632867 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:11.633466 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:11.633474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:11.633805 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 19:42:11.634041 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 19:42:11.634386 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":296,"threads_started":1}
I20260926 19:42:11.637051 16240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:42:11 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
I20260926 19:42:11.651371 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.637767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51740)
0926 19:42:11.637933 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.637952 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:11.638041 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:11.639013 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:11.639018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:11.639406 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:11.639570 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:11.639579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.640826 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.640831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:11.641773 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:11.641784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.642798 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.642802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:11.642956 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:11.643435 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:11.643450 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:11.645360 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:11.649014 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:11.649018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:11.649020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:11.649394 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:11.649809 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:11.649813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:11.649815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:11.649970 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:11.650674 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:11.650678 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:11.650808 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 19:42:11.650919 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 19:42:11.651111 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 19:42:11.651734 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.638024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.2.164.125:35187)
0926 19:42:11.638617 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:11.638623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:11.638647 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:42:11.638685 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:11.638691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:11.638856 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:11.638965 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:11.639716 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.640697 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.641908 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.642647 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.642757 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:11.645556 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:11.645569 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:11.645584 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:11.645617 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:11.648890 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:11.649528 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:11.649536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:11.649540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:11.649676 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:11.650092 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:11.650096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:11.650100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:11.650527 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:11.650740 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:11.651122 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 19:42:11.651374 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":326,"threads_started":1}
I20260926 19:42:11.654047 16240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:42:11.655369 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.654861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52136)
0926 19:42:11.655034 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.655228 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 19:42:11.655625 16189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
I20260926 19:42:11.657023 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.656717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52144)
0926 19:42:11.656831 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.656909 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 19:42:11.657249 16240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:42795: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab.
Entry for principal kudu/127.2.164.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab.
I20260926 19:42:11.661957 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.124:42795
--webserver_interface=127.2.164.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab
--principal=kudu/127.2.164.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:33707,127.2.164.125:35187,127.2.164.124:42795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:42:11.706545 16240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } attempt: 1
I20260926 19:42:11.707710 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.707255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52158)
0926 19:42:11.707458 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.707562 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 19:42:11.708673 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.708354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52168)
0926 19:42:11.708506 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.708571 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 19:42:11.708868 16240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:42795: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
I20260926 19:42:11.777077 16240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } attempt: 2
I20260926 19:42:11.778194 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.777719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52170)
0926 19:42:11.777919 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.778018 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 19:42:11.779361 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.778954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52174)
0926 19:42:11.779147 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.779235 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 19:42:11.779565 16240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:42795: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
I20260926 19:42:11.843439 16166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } attempt: 5
Sep 26 19:42:11 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
I20260926 19:42:11.855614 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.844231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51744 (local address 127.2.164.125:35187)
0926 19:42:11.844445 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:11.844451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:11.844470 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:42:11.844653 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:11.844657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:11.844732 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:11.844812 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:11.845642 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.846350 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.847460 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.847904 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.848008 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:11.850864 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:11.850869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:11.850872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:11.850902 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:11.853301 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:11.853901 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:11.853907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:11.853910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:11.854025 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:11.854354 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:11.854359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:11.854362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:11.854622 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:11.854745 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:11.855102 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 19:42:11.855296 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 19:42:11.855612 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.844161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51744)
0926 19:42:11.844371 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.844392 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:11.844527 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:11.844876 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:11.844881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:11.845289 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:11.845497 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:11.845508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.846477 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.846481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:11.847315 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:11.847337 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:11.848035 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:11.848041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:11.848223 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:11.848772 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:11.848789 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:11.850694 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:11.853442 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:11.853447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:11.853452 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:11.853793 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:11.854122 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:11.854127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:11.854131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:11.854255 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:11.854727 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:11.854731 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:11.854884 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 19:42:11.854996 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 19:42:11.855252 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 19:42:11.856753 16166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:42:11.857803 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.857454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52176)
0926 19:42:11.857604 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.857675 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 19:42:11.858729 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.858420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52186)
0926 19:42:11.858566 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.858636 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 19:42:11.858927 16166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:42795: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
I20260926 19:42:11.870765 16240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } attempt: 3
I20260926 19:42:11.871675 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.871268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52188)
0926 19:42:11.871452 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.871532 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 19:42:11.872761 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.872330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52192)
0926 19:42:11.872553 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.872628 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260926 19:42:11.872967 16240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:42795: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
I20260926 19:42:11.910135 16166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } attempt: 1
I20260926 19:42:11.911294 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.910816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52204)
0926 19:42:11.911039 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.911143 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 19:42:11.912321 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.911974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52206)
0926 19:42:11.912123 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.912193 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 19:42:11.912524 16166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:42795: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
W20260926 19:42:11.966480 16247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:11.966756 16247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:11.966883 16247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:11.973878 16247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:42:11.973987 16247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:11.974067 16247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:11.974139 16247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:42:11.974203 16247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:42:11.980768 16166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } attempt: 2
I20260926 19:42:11.981855 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.981378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52222)
0926 19:42:11.981593 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.981678 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 19:42:11.982849 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:11.982481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52234)
0926 19:42:11.982643 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:11.982740 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 19:42:11.983068 16166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:42795: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
I20260926 19:42:11.984187 16247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:33707,127.2.164.125:35187,127.2.164.124:42795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab
--principal=kudu/127.2.164.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.124:42795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.164.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:11.985499 16247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:11.986722 16247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:11.997982 16253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:11.998136 16252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:11.998153 16255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:11.998900 16247 server_base.cc:1034] running on GCE node
I20260926 19:42:11.999557 16247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:12.000880 16247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:12.002053 16247 hybrid_clock.cc:648] HybridClock initialized: now 1790451732001991 us; error 58 us; skew 500 ppm
I20260926 19:42:12.002388 16247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:11 dist-test-slave-mrw4 krb5kdc[16086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:12.005259 16247 init.cc:377] Logged in from keytab as kudu/127.2.164.124@KRBTEST.COM (short username kudu)
I20260926 19:42:12.006640 16247 webserver.cc:460] Webserver started at http://127.2.164.124:38257/ using document root <none> and password file <none>
I20260926 19:42:12.007272 16247 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:12.007396 16247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:12.007673 16247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:12.010528 16247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/data/instance:
uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1"
format_stamp: "Formatted at 2026-09-26 19:42:12 on dist-test-slave-mrw4"
server_key: "d559d023eb7c463c0621dfa2f0b1cfc1"
server_key_iv: "a4c97cd35a5a7b9ccf51feab2c63aa55"
server_key_version: "encryptionkey@0"
I20260926 19:42:12.011267 16247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/wal/instance:
uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1"
format_stamp: "Formatted at 2026-09-26 19:42:12 on dist-test-slave-mrw4"
server_key: "d559d023eb7c463c0621dfa2f0b1cfc1"
server_key_iv: "a4c97cd35a5a7b9ccf51feab2c63aa55"
server_key_version: "encryptionkey@0"
I20260926 19:42:12.016080 16247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:42:12.016167 16240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } attempt: 4
I20260926 19:42:12.017151 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.016699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52242)
0926 19:42:12.016914 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:12.016997 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 19:42:12.018121 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.017816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52244)
0926 19:42:12.017954 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:12.018022 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 19:42:12.018304 16240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:42795: Network error: Client connection negotiation failed: client connection to 127.2.164.124:42795: connect: Connection refused (error 111)
I20260926 19:42:12.019491 16262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:12.020756 16247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:42:12.020900 16247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/wal
uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1"
format_stamp: "Formatted at 2026-09-26 19:42:12 on dist-test-slave-mrw4"
server_key: "d559d023eb7c463c0621dfa2f0b1cfc1"
server_key_iv: "a4c97cd35a5a7b9ccf51feab2c63aa55"
server_key_version: "encryptionkey@0"
I20260926 19:42:12.021071 16247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:12.050374 16247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:12.054502 16247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:12.054755 16247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:12.071365 16247 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.124:42795
I20260926 19:42:12.071388 16313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.124:42795 every 8 connection(s)
I20260926 19:42:12.072638 16247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
I20260926 19:42:12.076110 16314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:12.080114 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 16247
I20260926 19:42:12.080322 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790451540687502-2705-0/minicluster-data/master-2/wal/instance
I20260926 19:42:12.080741 2705 external_mini_cluster.cc:1428] Setting key ff73fa09c1566c162c0bf588da9be5eb
I20260926 19:42:12.083076 16314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:42:12.089254 16166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } attempt: 3
Sep 26 19:42:12 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
Sep 26 19:42:12 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
Sep 26 19:42:12 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
I20260926 19:42:12.102597 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.086772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.2.164.126:33707)
0926 19:42:12.086986 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:12.086991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:12.087557 (+ 566us) server_negotiation.cc:407] Connection header received
0926 19:42:12.087706 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:12.087711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:12.087786 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:12.087887 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:12.088975 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.089860 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.091666 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.092292 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.092383 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.096082 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:12.096086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:12.096089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:12.096118 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:12.099393 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.100278 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.100284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.100287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.100420 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.100838 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.100843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.100846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.101272 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:12.101416 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:12.102044 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 19:42:12.102246 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:42:12.103241 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.082513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.2.164.124:42795)
0926 19:42:12.083290 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:12.083301 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:12.083340 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:42:12.084937 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:12.084958 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:12.085396 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:12.085693 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:12.087183 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.088720 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.090343 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.091275 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.091442 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.095764 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:12.095776 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:12.095790 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:12.095824 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:12.098711 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.099558 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.099566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.099569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.099785 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.100225 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.100230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.100234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.100910 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:12.101208 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:12.101575 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 19:42:12.101978 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":556,"thread_start_us":340,"threads_started":1}
I20260926 19:42:12.103322 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.086731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:33707 (local address 127.0.0.1:47964)
0926 19:42:12.087422 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:12.087463 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:12.087599 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:12.087931 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:12.087935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:12.088495 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:12.088833 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:12.088843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.090056 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.090061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:12.091512 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:12.091526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.092681 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.092687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:12.092857 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.093454 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:12.093476 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:12.095913 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:12.099504 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:12.099515 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:12.099526 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:12.100151 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:12.100524 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:12.100529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:12.100532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:12.100725 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:12.101398 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:12.101405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:12.101586 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 19:42:12.101953 (+ 367us) client_negotiation.cc:240] Negotiation successful
0926 19:42:12.102238 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":44,"spinlock_wait_cycles":20352,"thread_start_us":324,"threads_started":1}
I20260926 19:42:12.104892 16314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:42:12.105818 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.090068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52268 (local address 127.2.164.124:42795)
0926 19:42:12.090626 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:12.090631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:12.090649 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:42:12.090687 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:12.090692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:12.090752 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:12.090869 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:12.091736 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.092824 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.094426 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.094852 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.094964 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.099066 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:12.099073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:12.099077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:12.099109 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:12.102806 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.103479 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.103484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.103487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.103628 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.104081 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.104085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.104088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.104409 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:12.104529 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:12.104927 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 19:42:12.105118 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":266,"threads_started":1}
I20260926 19:42:12.106256 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.089790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52268)
0926 19:42:12.089979 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:12.089997 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:12.090092 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:12.090917 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:12.090923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:12.091374 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:12.091559 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:12.091570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.092972 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.092978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:12.094301 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:12.094314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.095018 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.095024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:12.095195 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.096383 (+ 1188us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:12.096401 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:12.098865 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:12.102915 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:12.102919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:12.102923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:12.103342 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:12.103753 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:12.103758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:12.103772 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:12.103943 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:12.104511 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:12.104515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:12.104663 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 19:42:12.104816 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 19:42:12.105042 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 19:42:12.111708 16166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d: Bootstrap starting.
Sep 26 19:42:12 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
Sep 26 19:42:12 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:12.115247 16166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:12.116644 16166 log.cc:824] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:12.119653 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.105647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:35187 (local address 127.0.0.1:51756)
0926 19:42:12.105799 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:12.105820 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:12.105912 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:12.106550 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:12.106554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:12.106931 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:12.107098 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:12.107109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.108408 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.108413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:12.109648 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:12.109662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.110540 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.110546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:12.110734 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.111246 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:12.111274 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:12.113671 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:12.117126 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:12.117131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:12.117135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:12.117606 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:12.118069 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:12.118073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:12.118077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:12.118244 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:12.118797 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:12.118801 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:12.118947 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 19:42:12.119087 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 19:42:12.119344 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 19:42:12.119740 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.105996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.2.164.125:35187)
0926 19:42:12.106150 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:12.106155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:12.106173 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:42:12.106228 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:12.106233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:12.106307 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:12.106397 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:12.107260 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.108270 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.109811 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.110404 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.110513 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.113849 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:12.113855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:12.113860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:12.113898 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:12.116997 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.117741 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.117747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.117761 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.117924 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.118368 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.118372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.118375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.118673 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:12.118821 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:12.119206 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 19:42:12.119395 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 19:42:12.120079 16166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d: No bootstrap required, opened a new log
I20260926 19:42:12.120870 16314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:42:12.121382 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.106304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.2.164.126:33707)
0926 19:42:12.106438 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:12.106443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:12.106458 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:42:12.106531 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:12.106536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:12.106593 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:12.106661 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:12.108117 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.108978 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.110407 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.110892 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.110974 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.115392 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:12.115396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:12.115399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:12.115429 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:12.118359 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.119020 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.119024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.119027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.119171 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.119599 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.119603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.119606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.120041 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:12.120296 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:12.120855 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 19:42:12.121050 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 19:42:12 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
I20260926 19:42:12.130059 16166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } }
I20260926 19:42:12.130345 16166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:12.130481 16166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64c8012f6f394a9fa703b95465058d7d, State: Initialized, Role: FOLLOWER
I20260926 19:42:12.131050 16166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } }
Sep 26 19:42:12 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
I20260926 19:42:12.131875 16321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } } }
I20260926 19:42:12.132383 16321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:42:12.133143 16166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:42:12.135378 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.121641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52278)
0926 19:42:12.121790 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:12.121807 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:12.121897 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:12.122597 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:12.122602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:12.123020 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:12.123183 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:12.123193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.124367 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.124372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:12.125404 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:12.125416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.126238 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.126244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:12.126476 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.127251 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:12.127282 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:12.129667 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:12.133071 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:12.133076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:12.133080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:12.133475 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:12.133816 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:12.133821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:12.133825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:12.133973 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:12.134488 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:12.134493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:12.134636 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 19:42:12.134753 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 19:42:12.135039 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 19:42:12.136148 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.122023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.2.164.124:42795)
0926 19:42:12.122308 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:12.122313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:12.122328 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:42:12.122367 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:12.122372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:12.122459 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:12.122552 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:12.123310 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.124266 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.125522 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.126097 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.126212 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.129870 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:12.129877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:12.129882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:12.129926 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:12.132986 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.133570 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.133575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.133579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.133700 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.134073 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.134076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.134077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.134371 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:12.135709 (+ 1338us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:12.135807 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 19:42:12.135952 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260926 19:42:12.140026 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.123685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.2.164.125:35187)
0926 19:42:12.123854 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:12.123860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:12.123876 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:42:12.123930 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:12.123934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:12.123995 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:12.124082 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:12.125558 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.126508 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.127757 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.128252 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.128354 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.133039 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:12.133045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:12.133049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:12.133084 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:12.136376 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.137110 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.137115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.137118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.137295 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.137718 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.137723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.137726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.138047 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:12.138300 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:12.139394 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0926 19:42:12.139611 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:42:12.144577 16314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1: Bootstrap starting.
I20260926 19:42:12.148226 16314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:12.149580 16314 log.cc:824] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:12.150614 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:42:12.152295 16314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1: No bootstrap required, opened a new log
W20260926 19:42:12.153288 16332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:42:12.153472 16332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:42:12.143436 16099 master_runner.cc:186] Error getting master registration for 127.2.164.125:35187: OK, instance_id { permanent_uuid: "53315d8e48de404e87330859a61884a6" instance_seqno: 1790451731599326 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:42:12.161391 16099 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 19:42:12.161798 16314 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } }
I20260926 19:42:12.162195 16314 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:12.162308 16314 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c4e0e604c2451a9dbc0179b3d6e6e1, State: Initialized, Role: FOLLOWER
I20260926 19:42:12.163029 16314 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } }
I20260926 19:42:12.169967 16340 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } } }
I20260926 19:42:12.172140 16340 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:42:12.172030 16314 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:42:12.189366 16321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:42:12.189702 16321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } }
I20260926 19:42:12.190231 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.163774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.2.164.126:33707)
0926 19:42:12.163951 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:12.163957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:12.172676 (+ 8719us) server_negotiation.cc:407] Connection header received
0926 19:42:12.172882 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:12.172887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:12.172975 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:12.173068 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:12.175407 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.176525 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.178571 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.179276 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.179377 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.183048 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:12.183055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:12.183059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:12.183104 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:12.186618 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.187729 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.187735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.187739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.187909 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.188408 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.188414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.188417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.188806 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:12.189060 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:12.189647 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 19:42:12.189890 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:42:12.191359 16321 leader_election.cc:288] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53315d8e48de404e87330859a61884a6 (127.2.164.125:35187), b2c4e0e604c2451a9dbc0179b3d6e6e1 (127.2.164.124:42795)
I20260926 19:42:12.192625 16289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64c8012f6f394a9fa703b95465058d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" is_pre_election: true
I20260926 19:42:12.193308 16289 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64c8012f6f394a9fa703b95465058d7d in term 0.
I20260926 19:42:12.194090 16115 leader_election.cc:302] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64c8012f6f394a9fa703b95465058d7d, b2c4e0e604c2451a9dbc0179b3d6e6e1; no voters:
I20260926 19:42:12.194640 16321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:42:12.194779 16321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:42:12.194860 16321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:12.198877 16321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } }
I20260926 19:42:12.200915 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.163418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.2.164.124:42795)
0926 19:42:12.163610 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:12.163616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:12.173800 (+ 10184us) server_negotiation.cc:407] Connection header received
0926 19:42:12.174276 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:12.174282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:12.174381 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:12.174451 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:12.179878 (+ 5427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.180908 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.185898 (+ 4990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.186590 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.186693 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.194364 (+ 7671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:12.194371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:12.194375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:12.194413 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:12.197688 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.198503 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.198508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.198512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.198688 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.199105 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.199110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.199114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.199483 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:12.199742 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:12.200334 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 19:42:12.200553 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:42:12.201752 16289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64c8012f6f394a9fa703b95465058d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1"
I20260926 19:42:12.201982 16289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:12.203953 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.164860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.2.164.125:35187)
0926 19:42:12.165031 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:12.165037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:12.177121 (+ 12084us) server_negotiation.cc:407] Connection header received
0926 19:42:12.177566 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:12.177572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:12.177680 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:12.177749 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:12.182978 (+ 5229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.184001 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.193488 (+ 9487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.194157 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.194286 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.197594 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:12.197601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:12.197606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:12.197655 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:12.201135 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.202060 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.202065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.202069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.202212 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.202553 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.202558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.202562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.202890 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:12.203113 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:12.203423 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 19:42:12.203635 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:42:12.205291 16321 leader_election.cc:288] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [CANDIDATE]: Term 1 election: Requested vote from peers 53315d8e48de404e87330859a61884a6 (127.2.164.125:35187), b2c4e0e604c2451a9dbc0179b3d6e6e1 (127.2.164.124:42795)
I20260926 19:42:12.205451 16289 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64c8012f6f394a9fa703b95465058d7d in term 1.
I20260926 19:42:12.205909 16115 leader_election.cc:302] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64c8012f6f394a9fa703b95465058d7d, b2c4e0e604c2451a9dbc0179b3d6e6e1; no voters:
I20260926 19:42:12.206764 16321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:42:12.209213 16321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [term 1 LEADER]: Becoming Leader. State: Replica: 64c8012f6f394a9fa703b95465058d7d, State: Running, Role: LEADER
I20260926 19:42:12.210215 16321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } }
I20260926 19:42:12.227599 16349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64c8012f6f394a9fa703b95465058d7d. Latest consensus state: current_term: 1 leader_uuid: "64c8012f6f394a9fa703b95465058d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } } }
I20260926 19:42:12.228061 16349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:12.229269 16354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:42:12.231904 16354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:42:12.242355 16289 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [term 1 FOLLOWER]: Refusing update from remote peer 64c8012f6f394a9fa703b95465058d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:42:12.243278 16349 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 19:42:12.253422 16356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:42:12.253705 16356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:42:12.241715 16247 master_runner.cc:186] Error getting master registration for 127.2.164.125:35187: OK, instance_id { permanent_uuid: "53315d8e48de404e87330859a61884a6" instance_seqno: 1790451731599326 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:42:12.259514 16247 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 19:42:12.271965 16340 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64c8012f6f394a9fa703b95465058d7d. Latest consensus state: current_term: 1 leader_uuid: "64c8012f6f394a9fa703b95465058d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } } }
I20260926 19:42:12.272258 16340 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:42:12.272938 16357 mvcc.cc:205] Tried to move back new op lower bound from 7333690295253942272 to 7333690295141740544. Current Snapshot: MvccSnapshot[applied={T|T < 7333690295253942272}]
I20260926 19:42:12.274484 16359 mvcc.cc:205] Tried to move back new op lower bound from 7333690295253942272 to 7333690295141740544. Current Snapshot: MvccSnapshot[applied={T|T < 7333690295253942272}]
I20260926 19:42:12.275305 16321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64c8012f6f394a9fa703b95465058d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } } }
I20260926 19:42:12.275626 16321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:12.277534 16340 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64c8012f6f394a9fa703b95465058d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } } }
I20260926 19:42:12.277755 16340 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:42:12.282388 16354 catalog_manager.cc:1347] Generated new cluster ID: a10adda537aa41bfa7904bcc50e7c0a9
I20260926 19:42:12.282487 16354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:42:12.295872 16354 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:42:12.297664 16354 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:42:12.310628 16354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d: Generated new TSK 0
I20260926 19:42:12.311589 16354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:42:12.317548 16240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } attempt: 5
Sep 26 19:42:12 dist-test-slave-mrw4 krb5kdc[16086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
I20260926 19:42:12.338660 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.318516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:42795 (local address 127.0.0.1:52290)
0926 19:42:12.318767 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:12.318785 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:12.318884 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:12.320031 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:12.320035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:12.320545 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:12.320737 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:12.320750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.322172 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.322177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:12.323523 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:12.323537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.324706 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.324712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:12.324911 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.325553 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:12.325576 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:12.328603 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:12.332767 (+ 4164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:12.332773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:12.332778 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:12.333612 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:12.334413 (+ 801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:12.334419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:12.334423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:12.334670 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:12.335722 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:12.335728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:12.335982 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 19:42:12.336180 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 19:42:12.337150 (+ 970us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260926 19:42:12.339341 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:12.319593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.2.164.124:42795)
0926 19:42:12.319753 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:12.319768 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:12.319785 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:42:12.319831 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:12.319836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:12.319907 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:12.319991 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:12.320906 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.322029 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.323688 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:12.324384 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:12.324496 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:12.328827 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:12.328834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:12.328838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:12.328875 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:12.332572 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.333807 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.333814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.333818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.334214 (+ 396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:12.334882 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:12.334887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:12.334891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:12.335530 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:12.336656 (+ 1126us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:12.337079 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 19:42:12.337958 (+ 879us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:42:12.345127 16240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6: Bootstrap starting.
I20260926 19:42:12.349597 16240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:12.351585 16240 log.cc:824] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:12.364319 16240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6: No bootstrap required, opened a new log
I20260926 19:42:12.379966 16240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } }
I20260926 19:42:12.380400 16240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:12.380534 16240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53315d8e48de404e87330859a61884a6, State: Initialized, Role: FOLLOWER
I20260926 19:42:12.381295 16240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } }
I20260926 19:42:12.388212 16368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } } }
I20260926 19:42:12.388609 16368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:42:12.390422 16240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:42:12.390777 16213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:12.394716 16213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [term 1 FOLLOWER]: Refusing update from remote peer 64c8012f6f394a9fa703b95465058d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:42:12.395512 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64c8012f6f394a9fa703b95465058d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53315d8e48de404e87330859a61884a6" is_pre_election: true
I20260926 19:42:12.395655 16349 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:42:12.395974 16215 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64c8012f6f394a9fa703b95465058d7d in term 1.
I20260926 19:42:12.396656 16214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64c8012f6f394a9fa703b95465058d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53315d8e48de404e87330859a61884a6"
I20260926 19:42:12.400040 16214 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64c8012f6f394a9fa703b95465058d7d in term 1.
I20260926 19:42:12.450510 16368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64c8012f6f394a9fa703b95465058d7d. Latest consensus state: current_term: 1 leader_uuid: "64c8012f6f394a9fa703b95465058d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } } }
I20260926 19:42:12.450826 16368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:42:12.451072 16377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64c8012f6f394a9fa703b95465058d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } } }
I20260926 19:42:12.451512 16377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:42:12.467943 16384 catalog_manager.cc:1259] Loaded cluster ID: a10adda537aa41bfa7904bcc50e7c0a9
I20260926 19:42:12.468109 16384 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6: loading cluster ID for follower catalog manager: success
I20260926 19:42:12.471885 16384 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6: acquiring CA information for follower catalog manager: success
I20260926 19:42:12.474277 16384 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53315d8e48de404e87330859a61884a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:42:12.939766 16349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64c8012f6f394a9fa703b95465058d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64c8012f6f394a9fa703b95465058d7d" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 33707 } } peers { permanent_uuid: "53315d8e48de404e87330859a61884a6" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 35187 } } peers { permanent_uuid: "b2c4e0e604c2451a9dbc0179b3d6e6e1" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 42795 } } }
I20260926 19:42:12.940064 16349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c8012f6f394a9fa703b95465058d7d [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:13.255553 16356 catalog_manager.cc:1259] Loaded cluster ID: a10adda537aa41bfa7904bcc50e7c0a9
I20260926 19:42:13.255647 16356 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1: loading cluster ID for follower catalog manager: success
I20260926 19:42:13.258090 16356 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1: acquiring CA information for follower catalog manager: success
I20260926 19:42:13.259835 16356 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2c4e0e604c2451a9dbc0179b3d6e6e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:42:13.459476 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 16099
I20260926 19:42:13.461107 16161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:13.570891 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 16099
W20260926 19:42:13.588130 16190 connection.cc:535] server connection from 127.0.0.1:51744 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 19:42:13.588294 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 16173
I20260926 19:42:13.589378 16235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:13.694126 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 16173
I20260926 19:42:13.710282 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 16247
I20260926 19:42:13.711354 16309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:13.815003 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 16247
2026-09-26T19:42:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4273 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:42:13 dist-test-slave-mrw4 krb5kdc[16394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:42:13 dist-test-slave-mrw4 krb5kdc[16394](info): set up 2 sockets
Sep 26 19:42:13 dist-test-slave-mrw4 krb5kdc[16394](info): commencing operation
krb5kdc: starting...
W20260926 19:42:14.821491 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:42:14 dist-test-slave-mrw4 krb5kdc[16394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:42:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:42:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:42:15.028172 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:41887
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:41887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:15.333072 16407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:15.333359 16407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:15.333467 16407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:15.340409 16407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:42:15.340498 16407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:15.340566 16407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:15.340641 16407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:42:15.340698 16407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:42:15.350044 16407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:41887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:41887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:15.351236 16407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:15.352456 16407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:15.363363 16412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:15.363324 16415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:15.363402 16413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:15.364292 16407 server_base.cc:1034] running on GCE node
I20260926 19:42:15.364976 16407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:15.366588 16407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:15.367801 16407 hybrid_clock.cc:648] HybridClock initialized: now 1790451735367726 us; error 58 us; skew 500 ppm
I20260926 19:42:15.368170 16407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:15 dist-test-slave-mrw4 krb5kdc[16394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451735, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:15.371057 16407 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:42:15.372480 16407 webserver.cc:460] Webserver started at http://127.2.164.126:40871/ using document root <none> and password file <none>
I20260926 19:42:15.373097 16407 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:15.373224 16407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:15.373502 16407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:15.376222 16407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "015593a6e47a4d299953c79e6677245b"
format_stamp: "Formatted at 2026-09-26 19:42:15 on dist-test-slave-mrw4"
server_key: "584adbeda53843e9ef8aab4399f78119"
server_key_iv: "dad06f088086e4465882f63dcd56b432"
server_key_version: "encryptionkey@0"
I20260926 19:42:15.376912 16407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "015593a6e47a4d299953c79e6677245b"
format_stamp: "Formatted at 2026-09-26 19:42:15 on dist-test-slave-mrw4"
server_key: "584adbeda53843e9ef8aab4399f78119"
server_key_iv: "dad06f088086e4465882f63dcd56b432"
server_key_version: "encryptionkey@0"
I20260926 19:42:15.381589 16407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 19:42:15.384742 16422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:15.386042 16407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:42:15.386188 16407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "015593a6e47a4d299953c79e6677245b"
format_stamp: "Formatted at 2026-09-26 19:42:15 on dist-test-slave-mrw4"
server_key: "584adbeda53843e9ef8aab4399f78119"
server_key_iv: "dad06f088086e4465882f63dcd56b432"
server_key_version: "encryptionkey@0"
I20260926 19:42:15.386349 16407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:15.404008 16407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:15.407847 16407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:15.408113 16407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:15.424897 16407 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:41887
I20260926 19:42:15.424903 16473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:41887 every 8 connection(s)
I20260926 19:42:15.426177 16407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:42:15.427229 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 16407
I20260926 19:42:15.427456 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:42:15.428241 2705 external_mini_cluster.cc:1428] Setting key 7260f1c78f1269c3c5a08169b3ddab33
I20260926 19:42:15.430665 16474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:15.440672 16474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b: Bootstrap starting.
Sep 26 19:42:15 dist-test-slave-mrw4 krb5kdc[16394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:15.443988 16474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:15.445003 16474 log.cc:824] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:15.447804 16474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b: No bootstrap required, opened a new log
I20260926 19:42:15.449067 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:15.430073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.2.164.126:41887)
0926 19:42:15.430833 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:15.430845 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:15.430886 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:42:15.432478 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:15.432499 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:15.432864 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:15.433256 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:15.434715 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:15.435993 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:15.437291 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:15.437993 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:15.438178 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:15.442371 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:15.442380 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:15.442393 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:15.442431 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:15.445379 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:15.446126 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:15.446133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:15.446138 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:15.446294 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:15.446654 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:15.446657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:15.446659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:15.447112 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:15.447310 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:15.447747 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 19:42:15.448107 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":340,"threads_started":1}
I20260926 19:42:15.453656 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:42:15.458563 16474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015593a6e47a4d299953c79e6677245b" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41887 } }
I20260926 19:42:15.458979 16474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:15.459062 16474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015593a6e47a4d299953c79e6677245b, State: Initialized, Role: FOLLOWER
I20260926 19:42:15.459628 16474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015593a6e47a4d299953c79e6677245b" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41887 } }
I20260926 19:42:15.459864 16474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:42:15.459964 16474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:42:15.460103 16474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:15.463094 16474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015593a6e47a4d299953c79e6677245b" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41887 } }
I20260926 19:42:15.463644 16474 leader_election.cc:302] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 015593a6e47a4d299953c79e6677245b; no voters:
I20260926 19:42:15.464283 16474 leader_election.cc:288] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:42:15.464372 16479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:42:15.465945 16479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [term 1 LEADER]: Becoming Leader. State: Replica: 015593a6e47a4d299953c79e6677245b, State: Running, Role: LEADER
I20260926 19:42:15.466445 16479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015593a6e47a4d299953c79e6677245b" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41887 } }
I20260926 19:42:15.469058 16474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:42:15.468984 16480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "015593a6e47a4d299953c79e6677245b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015593a6e47a4d299953c79e6677245b" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41887 } } }
I20260926 19:42:15.469389 16480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:15.469902 16481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 015593a6e47a4d299953c79e6677245b. Latest consensus state: current_term: 1 leader_uuid: "015593a6e47a4d299953c79e6677245b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015593a6e47a4d299953c79e6677245b" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41887 } } }
I20260926 19:42:15.470170 16481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:15.470178 16483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:42:15.475255 16483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:42:15.484472 16483 catalog_manager.cc:1347] Generated new cluster ID: 509e9875adec42f2ac4cca6c7de8f90b
I20260926 19:42:15.484570 16483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:42:15.499207 16483 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:42:15.500396 16483 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:42:15.510426 16483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 015593a6e47a4d299953c79e6677245b: Generated new TSK 0
I20260926 19:42:15.511238 16483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:42:15.520457 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 16407
I20260926 19:42:15.521497 16469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:15.608317 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 16407
2026-09-26T19:42:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1789 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:42:15 dist-test-slave-mrw4 krb5kdc[16502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:42:15 dist-test-slave-mrw4 krb5kdc[16502](info): set up 2 sockets
Sep 26 19:42:15 dist-test-slave-mrw4 krb5kdc[16502](info): commencing operation
krb5kdc: starting...
W20260926 19:42:16.613461 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:42:16 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:42:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:42:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:42:16.820299 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36815
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:36815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:17.129447 16515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:17.129737 16515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:17.129873 16515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:17.136866 16515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:42:17.136988 16515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:17.137072 16515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:17.137143 16515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:42:17.137220 16515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:42:17.146888 16515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:36815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:17.148162 16515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:17.149446 16515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:17.160573 16520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:17.160497 16521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:17.160529 16523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:17.161442 16515 server_base.cc:1034] running on GCE node
I20260926 19:42:17.162168 16515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:17.163487 16515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:17.164685 16515 hybrid_clock.cc:648] HybridClock initialized: now 1790451737164635 us; error 40 us; skew 500 ppm
I20260926 19:42:17.164991 16515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:17 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451737, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:17.172807 16515 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:42:17.174369 16515 webserver.cc:460] Webserver started at http://127.2.164.126:37967/ using document root <none> and password file <none>
I20260926 19:42:17.175029 16515 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:17.175144 16515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:17.175428 16515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:17.178316 16515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "85c29cdd8db244718bc1f1ea9abc2d49"
format_stamp: "Formatted at 2026-09-26 19:42:17 on dist-test-slave-mrw4"
server_key: "36c3066f4bf156a568657f0a1d3a2905"
server_key_iv: "e9612eb45dacc68ceead685168432d4a"
server_key_version: "encryptionkey@0"
I20260926 19:42:17.179049 16515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "85c29cdd8db244718bc1f1ea9abc2d49"
format_stamp: "Formatted at 2026-09-26 19:42:17 on dist-test-slave-mrw4"
server_key: "36c3066f4bf156a568657f0a1d3a2905"
server_key_iv: "e9612eb45dacc68ceead685168432d4a"
server_key_version: "encryptionkey@0"
I20260926 19:42:17.183986 16515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 19:42:17.187282 16530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:17.188656 16515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:42:17.188925 16515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "85c29cdd8db244718bc1f1ea9abc2d49"
format_stamp: "Formatted at 2026-09-26 19:42:17 on dist-test-slave-mrw4"
server_key: "36c3066f4bf156a568657f0a1d3a2905"
server_key_iv: "e9612eb45dacc68ceead685168432d4a"
server_key_version: "encryptionkey@0"
I20260926 19:42:17.189213 16515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:17.213835 16515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:17.218051 16515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:17.218315 16515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:17.235947 16515 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:36815
I20260926 19:42:17.235952 16581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:36815 every 8 connection(s)
I20260926 19:42:17.237375 16515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:42:17.239372 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 16515
I20260926 19:42:17.239588 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:42:17.240408 2705 external_mini_cluster.cc:1428] Setting key 1ce92c4561db7c8f424f55203710032f
I20260926 19:42:17.241705 16582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:42:17 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:17.255961 16582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Bootstrap starting.
I20260926 19:42:17.259431 16582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:17.260525 16582 log.cc:824] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:17.263686 16582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: No bootstrap required, opened a new log
I20260926 19:42:17.264143 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:17.242422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51292 (local address 127.2.164.126:36815)
0926 19:42:17.243141 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:17.243154 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:17.243207 (+ 53us) server_negotiation.cc:407] Connection header received
0926 19:42:17.245314 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:17.245341 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:17.245836 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:17.246284 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:17.247978 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:17.249757 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:17.251521 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:17.252198 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:17.252331 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:17.257105 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:17.257113 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:17.257127 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:17.257187 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:17.260110 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:17.260883 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:17.260890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:17.260896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:17.261085 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:17.261514 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:17.261519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:17.261522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:17.262118 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:17.262347 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:17.262784 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 19:42:17.263140 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":318,"threads_started":1}
I20260926 19:42:17.270841 16582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:17.271163 16582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:17.271250 16582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c29cdd8db244718bc1f1ea9abc2d49, State: Initialized, Role: FOLLOWER
I20260926 19:42:17.271759 16582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:17.271970 16582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:42:17.272089 16582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:42:17.272224 16582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:17.274295 16582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:17.274751 16582 leader_election.cc:302] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c29cdd8db244718bc1f1ea9abc2d49; no voters:
I20260926 19:42:17.275285 16582 leader_election.cc:288] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:42:17.275410 16587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:42:17.276000 16587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 1 LEADER]: Becoming Leader. State: Replica: 85c29cdd8db244718bc1f1ea9abc2d49, State: Running, Role: LEADER
I20260926 19:42:17.276490 16587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:17.277143 16582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:42:17.279024 16589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85c29cdd8db244718bc1f1ea9abc2d49. Latest consensus state: current_term: 1 leader_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } } }
I20260926 19:42:17.279367 16589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:17.281329 16588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } } }
I20260926 19:42:17.281598 16588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:17.283066 16594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:42:17.288723 16594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:42:17.297859 16594 catalog_manager.cc:1347] Generated new cluster ID: c3cbdcfb34be4b69b71a895bd9253179
I20260926 19:42:17.297966 16594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:42:17.306686 16594 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:42:17.307898 16594 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:42:17.315094 16594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Generated new TSK 0
I20260926 19:42:17.315979 16594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:42:17.378196 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36815
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:17.689705 16608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:17.689960 16608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:17.690073 16608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:17.697273 16608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:17.697376 16608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:17.697523 16608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:42:17.707175 16608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:17.708443 16608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:17.709765 16608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:17.722182 16614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:17.722204 16613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:17.722205 16616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:17.723030 16608 server_base.cc:1034] running on GCE node
I20260926 19:42:17.723573 16608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:17.724443 16608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:17.725651 16608 hybrid_clock.cc:648] HybridClock initialized: now 1790451737725608 us; error 33 us; skew 500 ppm
I20260926 19:42:17.725960 16608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:17 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451737, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:17.728899 16608 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:42:17.730355 16608 webserver.cc:460] Webserver started at http://127.2.164.65:38929/ using document root <none> and password file <none>
I20260926 19:42:17.730993 16608 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:17.731086 16608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:17.731335 16608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:17.734174 16608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "7c38a9fe2c584e36a383e6890a0f65b5"
format_stamp: "Formatted at 2026-09-26 19:42:17 on dist-test-slave-mrw4"
server_key: "67ffe3b2bac69e76772dcee1c33edabf"
server_key_iv: "357bb82541eed9f83161601d9c0b2631"
server_key_version: "encryptionkey@0"
I20260926 19:42:17.734864 16608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "7c38a9fe2c584e36a383e6890a0f65b5"
format_stamp: "Formatted at 2026-09-26 19:42:17 on dist-test-slave-mrw4"
server_key: "67ffe3b2bac69e76772dcee1c33edabf"
server_key_iv: "357bb82541eed9f83161601d9c0b2631"
server_key_version: "encryptionkey@0"
I20260926 19:42:17.739768 16608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 19:42:17.743053 16623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:17.744354 16608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:42:17.744503 16608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "7c38a9fe2c584e36a383e6890a0f65b5"
format_stamp: "Formatted at 2026-09-26 19:42:17 on dist-test-slave-mrw4"
server_key: "67ffe3b2bac69e76772dcee1c33edabf"
server_key_iv: "357bb82541eed9f83161601d9c0b2631"
server_key_version: "encryptionkey@0"
I20260926 19:42:17.744675 16608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:17.759860 16608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:17.763609 16608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:17.763834 16608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:17.764640 16608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:17.765969 16608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:17.766041 16608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:17.766121 16608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:17.766165 16608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:17.794414 16608 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:35477
I20260926 19:42:17.794459 16735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:35477 every 8 connection(s)
I20260926 19:42:17.795612 16608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:42:17.797878 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 16608
I20260926 19:42:17.798080 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:42:17.798825 2705 external_mini_cluster.cc:1428] Setting key 4dd5c99890ecb45c5d07e4cbe914f095
Sep 26 19:42:17 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451737, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:17.815557 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:17.798150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:57129 (local address 127.2.164.126:36815)
0926 19:42:17.798602 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:17.798609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:17.799694 (+ 1085us) server_negotiation.cc:407] Connection header received
0926 19:42:17.801616 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:17.801622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:17.801701 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:17.801802 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:17.803279 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:17.803989 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:17.805519 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:17.806103 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:17.806221 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:17.810063 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:17.810069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:17.810072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:17.810106 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:17.812369 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:17.813282 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:17.813288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:17.813292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:17.813421 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:17.813854 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:17.813857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:17.813859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:17.814139 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:17.814279 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:17.815029 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 19:42:17.815216 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":203,"threads_started":1}
I20260926 19:42:17.816721 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:17.798756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:57129)
0926 19:42:17.799534 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:17.799585 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:17.801305 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:17.802040 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:17.802048 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:17.802574 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:17.803115 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:17.803133 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:17.804130 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:17.804133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:17.805392 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:17.805404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:17.806333 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:17.806340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:17.806596 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:17.807222 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:17.807250 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:17.809842 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:17.812500 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:17.812508 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:17.812523 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:17.813065 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:17.813519 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:17.813523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:17.813526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:17.813749 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:17.814247 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:17.814254 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:17.814654 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 19:42:17.814951 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 19:42:17.815434 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":299,"threads_started":1}
I20260926 19:42:17.818312 16736 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36815
I20260926 19:42:17.818586 16736 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:17.819260 16736 heartbeater.cc:511] Master 127.2.164.126:36815 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
I20260926 19:42:17.821251 16547 ts_manager.cc:194] Registered new tserver with Master: 7c38a9fe2c584e36a383e6890a0f65b5 (127.2.164.65:35477)
I20260926 19:42:17.822681 16547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:57129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:42:17.847883 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36815
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:18.152491 16742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:18.152740 16742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:18.152828 16742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:18.159606 16742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:18.159695 16742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:18.159798 16742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:42:18.168803 16742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:18.169925 16742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:18.171097 16742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:18.183107 16747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:18.183139 16750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:18.183079 16748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:18.184068 16742 server_base.cc:1034] running on GCE node
I20260926 19:42:18.184672 16742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:18.185590 16742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:18.186781 16742 hybrid_clock.cc:648] HybridClock initialized: now 1790451738186717 us; error 55 us; skew 500 ppm
I20260926 19:42:18.187105 16742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:18 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:18.190140 16742 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:42:18.191560 16742 webserver.cc:460] Webserver started at http://127.2.164.66:39735/ using document root <none> and password file <none>
I20260926 19:42:18.192221 16742 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:18.192350 16742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:18.192618 16742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:18.195482 16742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "9c78e3a48b444dddbc7b7074a0a811d2"
format_stamp: "Formatted at 2026-09-26 19:42:18 on dist-test-slave-mrw4"
server_key: "27d81f262e8ef9dc9a854023c930b0e2"
server_key_iv: "4831d34ba12abd5e5a95d8477358328f"
server_key_version: "encryptionkey@0"
I20260926 19:42:18.196183 16742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "9c78e3a48b444dddbc7b7074a0a811d2"
format_stamp: "Formatted at 2026-09-26 19:42:18 on dist-test-slave-mrw4"
server_key: "27d81f262e8ef9dc9a854023c930b0e2"
server_key_iv: "4831d34ba12abd5e5a95d8477358328f"
server_key_version: "encryptionkey@0"
I20260926 19:42:18.201333 16742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 19:42:18.204787 16757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:18.206223 16742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:42:18.206377 16742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "9c78e3a48b444dddbc7b7074a0a811d2"
format_stamp: "Formatted at 2026-09-26 19:42:18 on dist-test-slave-mrw4"
server_key: "27d81f262e8ef9dc9a854023c930b0e2"
server_key_iv: "4831d34ba12abd5e5a95d8477358328f"
server_key_version: "encryptionkey@0"
I20260926 19:42:18.206561 16742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:18.230445 16742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:18.234149 16742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:18.234426 16742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:18.235348 16742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:18.236994 16742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:18.237087 16742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:18.237212 16742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:18.237283 16742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 19:42:18.266990 16742 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:35953
I20260926 19:42:18.267045 16869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:35953 every 8 connection(s)
I20260926 19:42:18.268168 16742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:42:18.276855 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 16742
I20260926 19:42:18.277021 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:42:18.277556 2705 external_mini_cluster.cc:1428] Setting key 0df2350c04a4d3f6b0af6a09e31a9ac8
Sep 26 19:42:18 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:18.286837 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:18.270697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:33111 (local address 127.2.164.126:36815)
0926 19:42:18.270870 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:18.270874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:18.271983 (+ 1109us) server_negotiation.cc:407] Connection header received
0926 19:42:18.273828 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:18.273833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:18.273925 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:18.274009 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:18.275439 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.276149 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.277850 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.278237 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.278347 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:18.281462 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:18.281467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:18.281471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:18.281502 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:18.283983 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:18.284664 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:18.284668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:18.284670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:18.284806 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:18.285224 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:18.285227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:18.285229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:18.285477 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:18.285607 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:18.286272 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 19:42:18.286500 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:42:18.287786 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:18.271136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:33111)
0926 19:42:18.271826 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:18.271868 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:18.273555 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:18.274252 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:18.274259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:18.274751 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:18.275277 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:18.275304 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.276292 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.276296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:18.277694 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:18.277706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.278440 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.278447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:18.278659 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:18.279237 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:18.279257 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:18.281268 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:18.284118 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:18.284127 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:18.284144 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:18.284542 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:18.284905 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:18.284910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:18.284914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:18.285086 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:18.285601 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:18.285608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:18.285910 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 19:42:18.286146 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 19:42:18.286661 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":266,"threads_started":1}
I20260926 19:42:18.289178 16870 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36815
I20260926 19:42:18.289469 16870 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:18.290097 16870 heartbeater.cc:511] Master 127.2.164.126:36815 requested a full tablet report, sending...
I20260926 19:42:18.291411 16547 ts_manager.cc:194] Registered new tserver with Master: 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953)
I20260926 19:42:18.292160 16547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:33111
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:42:18.322532 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36815
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:18.624541 16875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:18.624795 16875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:18.624886 16875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:18.631762 16875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:18.631850 16875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:18.631961 16875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:42:18.640980 16875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:18.642099 16875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:18.643266 16875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:18.654960 16881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:18.654991 16883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:18.654927 16880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:18.655750 16875 server_base.cc:1034] running on GCE node
I20260926 19:42:18.656282 16875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:18.657123 16875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:18.658324 16875 hybrid_clock.cc:648] HybridClock initialized: now 1790451738658278 us; error 42 us; skew 500 ppm
I20260926 19:42:18.658586 16875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:18 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:18.661428 16875 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:42:18.662675 16875 webserver.cc:460] Webserver started at http://127.2.164.67:38085/ using document root <none> and password file <none>
I20260926 19:42:18.663286 16875 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:18.663391 16875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:18.663645 16875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:18.666334 16875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "09f6d961f8e6451aac31f6c4a6ec16bd"
format_stamp: "Formatted at 2026-09-26 19:42:18 on dist-test-slave-mrw4"
server_key: "3551adf83dd4692e47805172033a97e6"
server_key_iv: "62863c39890006955542ab1b1ea5f168"
server_key_version: "encryptionkey@0"
I20260926 19:42:18.667016 16875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "09f6d961f8e6451aac31f6c4a6ec16bd"
format_stamp: "Formatted at 2026-09-26 19:42:18 on dist-test-slave-mrw4"
server_key: "3551adf83dd4692e47805172033a97e6"
server_key_iv: "62863c39890006955542ab1b1ea5f168"
server_key_version: "encryptionkey@0"
I20260926 19:42:18.671654 16875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 19:42:18.674798 16890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:18.676002 16875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 19:42:18.676134 16875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "09f6d961f8e6451aac31f6c4a6ec16bd"
format_stamp: "Formatted at 2026-09-26 19:42:18 on dist-test-slave-mrw4"
server_key: "3551adf83dd4692e47805172033a97e6"
server_key_iv: "62863c39890006955542ab1b1ea5f168"
server_key_version: "encryptionkey@0"
I20260926 19:42:18.676301 16875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:18.703253 16875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:18.707106 16875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:18.707351 16875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:18.708225 16875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:18.709589 16875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:18.709671 16875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:18.709780 16875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:18.709829 16875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:18.738641 16875 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:34783
I20260926 19:42:18.738686 17002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:34783 every 8 connection(s)
I20260926 19:42:18.739801 16875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:42:18.740554 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 16875
I20260926 19:42:18.740727 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:42:18.741292 2705 external_mini_cluster.cc:1428] Setting key 1f7b87d217fe43046daa7b582910bdcc
Sep 26 19:42:18 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451738, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:18.759030 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:18.742535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:38827 (local address 127.2.164.126:36815)
0926 19:42:18.742725 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:18.742730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:18.743954 (+ 1224us) server_negotiation.cc:407] Connection header received
0926 19:42:18.745909 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:18.745913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:18.745977 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:18.746054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:18.747385 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.748147 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.749746 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.750137 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.750226 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:18.753373 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:18.753377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:18.753381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:18.753412 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:18.756380 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:18.757073 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:18.757077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:18.757079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:18.757218 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:18.757554 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:18.757557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:18.757559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:18.757809 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:18.757912 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:18.758629 (+ 717us) server_negotiation.cc:299] Negotiation successful
0926 19:42:18.758784 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 19:42:18.759953 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:18.743059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:38827)
0926 19:42:18.743816 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:18.743863 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:18.745663 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:18.746221 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:18.746229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:18.746712 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:18.747260 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:18.747279 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.748287 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.748292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:18.749630 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:18.749642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.750351 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.750357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:18.750552 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:18.751110 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:18.751130 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:18.753147 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:18.756524 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:18.756534 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:18.756549 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:18.756970 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:18.757304 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:18.757307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:18.757310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:18.757476 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:18.757901 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:18.757906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:18.758299 (+ 393us) client_negotiation.cc:769] Sending connection context
0926 19:42:18.758532 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 19:42:18.758924 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":265,"threads_started":1}
I20260926 19:42:18.761149 17003 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36815
I20260926 19:42:18.761476 17003 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:18.762135 17003 heartbeater.cc:511] Master 127.2.164.126:36815 requested a full tablet report, sending...
I20260926 19:42:18.763459 16547 ts_manager.cc:194] Registered new tserver with Master: 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783)
I20260926 19:42:18.764207 16547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:38827
I20260926 19:42:18.768543 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:42:18.784292 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:18.772721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.2.164.126:36815)
0926 19:42:18.772876 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:18.772883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:18.773241 (+ 358us) server_negotiation.cc:407] Connection header received
0926 19:42:18.773443 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:18.773448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:18.773514 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:18.773589 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:18.774847 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.775550 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.776867 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.777352 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.777461 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:18.779590 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:18.779596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:18.779600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:18.779630 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:18.781903 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:18.782512 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:18.782518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:18.782521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:18.782651 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:18.783007 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:18.783012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:18.783014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:18.783291 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:18.783519 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:18.783810 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 19:42:18.784007 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 19:42:18.787883 16547 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:42:18.794898 16547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:42:18.819522 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:18.805690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:34783 (local address 127.0.0.1:43216)
0926 19:42:18.806539 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:18.806573 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:18.806740 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:18.812791 (+ 6051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:18.812800 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:18.812832 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:18.813318 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:18.813333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.814910 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.814916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:18.816814 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:18.816828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.818585 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.818591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:18.818792 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:18.818840 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 19:42:18.818975 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 19:42:18.819148 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":1,"thread_start_us":1252,"threads_started":1}
I20260926 19:42:18.819538 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:18.805994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.2.164.67:34783)
0926 19:42:18.811572 (+ 5578us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:18.811582 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:18.811617 (+ 35us) server_negotiation.cc:407] Connection header received
0926 19:42:18.811721 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:18.811730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:18.811987 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:18.812172 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:18.813604 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.814787 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.816988 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.818393 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.818511 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:18.818722 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:18.819054 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 19:42:18.819213 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5422,"thread_start_us":5277,"threads_started":1}
I20260926 19:42:18.820413 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:18.806548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.0.0.1:55090)
0926 19:42:18.810963 (+ 4415us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:18.810980 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:18.811076 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:18.813105 (+ 2029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:18.813114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:18.813139 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:18.813673 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:18.813686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.815782 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.815787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:18.817885 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:18.817897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.819648 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.819655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:18.819806 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:18.819836 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 19:42:18.819911 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 19:42:18.820021 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4296,"mutex_wait_us":24,"thread_start_us":3581,"threads_started":1}
I20260926 19:42:18.820474 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:18.807296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.2.164.65:35477)
0926 19:42:18.812524 (+ 5228us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:18.812533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:18.812560 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:42:18.812654 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:18.812662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:18.812890 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:18.813867 (+ 977us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:18.814094 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.815613 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.818028 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.819176 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.819300 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:18.819517 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:18.820008 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 19:42:18.820162 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5080,"thread_start_us":4946,"threads_started":1}
I20260926 19:42:18.821044 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:18.806017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:35953 (local address 127.0.0.1:49836)
0926 19:42:18.810377 (+ 4360us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:18.810397 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:18.810512 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:18.813336 (+ 2824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:18.813341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:18.813369 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:18.813913 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:18.813925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.815739 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.815748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:18.818718 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:18.818732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.820344 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.820350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:18.820512 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:18.820540 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 19:42:18.820604 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 19:42:18.820717 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4238,"mutex_wait_us":38,"thread_start_us":4681,"threads_started":1}
I20260926 19:42:18.822031 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:18.806863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.2.164.66:35953)
0926 19:42:18.812786 (+ 5923us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:18.812794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:18.812818 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:42:18.812899 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:18.812906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:18.813109 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:18.813495 (+ 386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:18.814074 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.815600 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.818863 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:18.820103 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:18.821359 (+ 1256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:18.821539 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:18.821603 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 19:42:18.821752 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5779,"thread_start_us":5694,"threads_started":1}
I20260926 19:42:18.823933 16671 tablet_service.cc:1467] Processing CreateTablet for tablet 04d279549e8242809df92a242a31d97d (DEFAULT_TABLE table=test-table [id=b6a4438407964f8b8a3e0903edf0b933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:18.823877 16938 tablet_service.cc:1467] Processing CreateTablet for tablet 04d279549e8242809df92a242a31d97d (DEFAULT_TABLE table=test-table [id=b6a4438407964f8b8a3e0903edf0b933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:18.825074 16671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d279549e8242809df92a242a31d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:18.825279 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d279549e8242809df92a242a31d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:18.825317 16736 heartbeater.cc:503] Master 127.2.164.126:36815 was elected leader, sending a full tablet report...
I20260926 19:42:18.825260 16805 tablet_service.cc:1467] Processing CreateTablet for tablet 04d279549e8242809df92a242a31d97d (DEFAULT_TABLE table=test-table [id=b6a4438407964f8b8a3e0903edf0b933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:18.826454 16805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d279549e8242809df92a242a31d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:18.836155 17021 tablet_bootstrap.cc:492] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: Bootstrap starting.
I20260926 19:42:18.838864 17022 tablet_bootstrap.cc:492] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: Bootstrap starting.
I20260926 19:42:18.839341 17021 tablet_bootstrap.cc:654] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:18.840581 17021 log.cc:824] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:18.842176 17022 tablet_bootstrap.cc:654] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:18.843348 17022 log.cc:824] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:18.843604 17021 tablet_bootstrap.cc:492] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: No bootstrap required, opened a new log
I20260926 19:42:18.843850 17021 ts_tablet_manager.cc:1397] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 19:42:18.845228 17023 tablet_bootstrap.cc:492] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: Bootstrap starting.
I20260926 19:42:18.846062 17022 tablet_bootstrap.cc:492] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: No bootstrap required, opened a new log
I20260926 19:42:18.846292 17022 ts_tablet_manager.cc:1397] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260926 19:42:18.847704 17023 tablet_bootstrap.cc:654] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:18.848644 17023 log.cc:824] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:18.850984 17023 tablet_bootstrap.cc:492] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: No bootstrap required, opened a new log
I20260926 19:42:18.851199 17023 ts_tablet_manager.cc:1397] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260926 19:42:18.853621 17021 raft_consensus.cc:348] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } }
I20260926 19:42:18.854123 17021 raft_consensus.cc:374] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:18.854219 17021 raft_consensus.cc:729] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c38a9fe2c584e36a383e6890a0f65b5, State: Initialized, Role: FOLLOWER
I20260926 19:42:18.854797 17021 consensus_queue.cc:260] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } }
I20260926 19:42:18.856374 17021 ts_tablet_manager.cc:1428] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 19:42:18.857134 17022 raft_consensus.cc:348] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } }
I20260926 19:42:18.857504 17022 raft_consensus.cc:374] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:18.857594 17022 raft_consensus.cc:729] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f6d961f8e6451aac31f6c4a6ec16bd, State: Initialized, Role: FOLLOWER
I20260926 19:42:18.858105 17022 consensus_queue.cc:260] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } }
I20260926 19:42:18.859026 17003 heartbeater.cc:503] Master 127.2.164.126:36815 was elected leader, sending a full tablet report...
I20260926 19:42:18.859371 17023 raft_consensus.cc:348] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } }
I20260926 19:42:18.859619 17022 ts_tablet_manager.cc:1428] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260926 19:42:18.859649 17023 raft_consensus.cc:374] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:18.859704 17023 raft_consensus.cc:729] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c78e3a48b444dddbc7b7074a0a811d2, State: Initialized, Role: FOLLOWER
I20260926 19:42:18.860169 17023 consensus_queue.cc:260] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } }
I20260926 19:42:18.861129 16870 heartbeater.cc:503] Master 127.2.164.126:36815 was elected leader, sending a full tablet report...
I20260926 19:42:18.861721 17023 ts_tablet_manager.cc:1428] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
W20260926 19:42:18.991366 17004 tablet.cc:2369] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:42:19.020150 16871 tablet.cc:2369] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:42:19.047808 16737 tablet.cc:2369] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:42:19.125111 17029 raft_consensus.cc:482] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:42:19.125458 17029 raft_consensus.cc:504] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } }
I20260926 19:42:19.127024 17029 leader_election.cc:288] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783), 7c38a9fe2c584e36a383e6890a0f65b5 (127.2.164.65:35477)
I20260926 19:42:19.135676 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:19.127460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:56441 (local address 127.2.164.67:34783)
0926 19:42:19.127650 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:19.127656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:19.128335 (+ 679us) server_negotiation.cc:407] Connection header received
0926 19:42:19.128529 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:19.128534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:19.128614 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:19.128712 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:19.129513 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.130604 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.133407 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.134609 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.134765 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:19.134839 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:19.135307 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 19:42:19.135432 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:42:19.135680 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:19.127643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:50081 (local address 127.2.164.65:35477)
0926 19:42:19.127831 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:19.127836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:19.128368 (+ 532us) server_negotiation.cc:407] Connection header received
0926 19:42:19.128542 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:19.128547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:19.128614 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:19.128708 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:19.129513 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.130603 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.133420 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.134609 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.134765 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:19.134839 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:19.135308 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 19:42:19.135451 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:42:19.135931 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:19.127352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.2.164.66:50081)
0926 19:42:19.128270 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:19.128285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:19.128433 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:19.128736 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:19.128741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:19.128794 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:19.129363 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:19.129376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.130772 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.130776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:19.133256 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:19.133265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.134862 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.134868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:19.135092 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:19.135179 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 19:42:19.135272 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 19:42:19.135445 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":149,"spinlock_wait_cycles":84864,"thread_start_us":463,"threads_started":1}
I20260926 19:42:19.135951 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:19.127288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:34783 (local address 127.2.164.66:56441)
0926 19:42:19.128191 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:19.128224 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:19.128392 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:19.128736 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:19.128753 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:19.128791 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:19.129365 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:19.129376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.130761 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.130766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:19.133221 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:19.133237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.134875 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.134880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:19.135075 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:19.135177 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 19:42:19.135272 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 19:42:19.135445 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":160,"spinlock_wait_cycles":1280,"thread_start_us":691,"threads_started":1}
I20260926 19:42:19.137054 16958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d279549e8242809df92a242a31d97d" candidate_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" is_pre_election: true
I20260926 19:42:19.137055 16691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d279549e8242809df92a242a31d97d" candidate_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" is_pre_election: true
I20260926 19:42:19.137617 16691 raft_consensus.cc:2454] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c78e3a48b444dddbc7b7074a0a811d2 in term 0.
I20260926 19:42:19.137617 16958 raft_consensus.cc:2454] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c78e3a48b444dddbc7b7074a0a811d2 in term 0.
I20260926 19:42:19.138358 16761 leader_election.cc:302] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c38a9fe2c584e36a383e6890a0f65b5, 9c78e3a48b444dddbc7b7074a0a811d2; no voters:
I20260926 19:42:19.138777 17029 raft_consensus.cc:2781] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:42:19.138906 17029 raft_consensus.cc:482] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:42:19.139001 17029 raft_consensus.cc:3037] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:19.141144 17029 raft_consensus.cc:504] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } }
I20260926 19:42:19.141716 17029 leader_election.cc:288] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [CANDIDATE]: Term 1 election: Requested vote from peers 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783), 7c38a9fe2c584e36a383e6890a0f65b5 (127.2.164.65:35477)
I20260926 19:42:19.142180 16691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d279549e8242809df92a242a31d97d" candidate_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c38a9fe2c584e36a383e6890a0f65b5"
I20260926 19:42:19.142176 16958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d279549e8242809df92a242a31d97d" candidate_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd"
I20260926 19:42:19.142374 16958 raft_consensus.cc:3037] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:19.142374 16691 raft_consensus.cc:3037] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:19.145390 16691 raft_consensus.cc:2454] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c78e3a48b444dddbc7b7074a0a811d2 in term 1.
I20260926 19:42:19.145398 16958 raft_consensus.cc:2454] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c78e3a48b444dddbc7b7074a0a811d2 in term 1.
I20260926 19:42:19.145780 16760 leader_election.cc:302] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d961f8e6451aac31f6c4a6ec16bd, 9c78e3a48b444dddbc7b7074a0a811d2; no voters:
I20260926 19:42:19.146039 17029 raft_consensus.cc:2781] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:42:19.146492 17029 raft_consensus.cc:686] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 1 LEADER]: Becoming Leader. State: Replica: 9c78e3a48b444dddbc7b7074a0a811d2, State: Running, Role: LEADER
I20260926 19:42:19.146930 17029 consensus_queue.cc:237] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } }
I20260926 19:42:19.151674 16547 catalog_manager.cc:5495] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66). New cstate: current_term: 1 leader_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } health_report { overall_health: UNKNOWN } } }
I20260926 19:42:19.177472 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:19.172280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.2.164.66:35953)
0926 19:42:19.172476 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:19.172482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:19.172500 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:42:19.172691 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:19.172695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:19.172769 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:19.172895 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:42:19.173268 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.173953 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.175354 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.175763 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.175867 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:19.176346 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:42:19.176888 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:42:19.177017 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:19.177110 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 19:42:19.177211 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:42:19.185334 16958 raft_consensus.cc:1261] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd [term 1 FOLLOWER]: Refusing update from remote peer 9c78e3a48b444dddbc7b7074a0a811d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:42:19.185331 16691 raft_consensus.cc:1261] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Refusing update from remote peer 9c78e3a48b444dddbc7b7074a0a811d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:42:19.186211 17029 consensus_queue.cc:1035] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:42:19.186549 17032 consensus_queue.cc:1035] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:42:19.196828 17037 mvcc.cc:205] Tried to move back new op lower bound from 7333690323693834240 to 7333690323549736960. Current Snapshot: MvccSnapshot[applied={T|T < 7333690323693834240}]
I20260926 19:42:19.200645 17035 mvcc.cc:205] Tried to move back new op lower bound from 7333690323693834240 to 7333690323549736960. Current Snapshot: MvccSnapshot[applied={T|T < 7333690323693834240}]
I20260926 19:42:19.213392 16547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:42:19.213621 16547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:42:19.216964 16547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Sending DeleteTablet for 3 replicas of tablet 04d279549e8242809df92a242a31d97d
I20260926 19:42:19.218091 16805 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d279549e8242809df92a242a31d97d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:19 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:49836
I20260926 19:42:19.218086 16938 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d279549e8242809df92a242a31d97d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:19 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:43216
I20260926 19:42:19.218694 16671 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d279549e8242809df92a242a31d97d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:19 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:55090
I20260926 19:42:19.219771 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 16608
I20260926 19:42:19.225697 17047 tablet_replica.cc:330] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: stopping tablet replica
I20260926 19:42:19.226150 17047 raft_consensus.cc:2229] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:42:19.226678 17047 raft_consensus.cc:2258] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:42:19.229739 17047 ts_tablet_manager.cc:1905] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:42:19.233907 17046 tablet_replica.cc:330] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: stopping tablet replica
I20260926 19:42:19.234340 17046 raft_consensus.cc:2229] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:42:19.235059 17046 raft_consensus.cc:2258] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:42:19.235780 17045 tablet_replica.cc:330] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: stopping tablet replica
I20260926 19:42:19.236267 17045 raft_consensus.cc:2229] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:42:19.236318 17047 ts_tablet_manager.cc:1918] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:42:19.236524 17047 log.cc:1196] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/04d279549e8242809df92a242a31d97d
I20260926 19:42:19.236801 17045 raft_consensus.cc:2258] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:42:19.237030 17047 ts_tablet_manager.cc:1939] T 04d279549e8242809df92a242a31d97d P 7c38a9fe2c584e36a383e6890a0f65b5: Deleting consensus metadata
I20260926 19:42:19.238678 16534 catalog_manager.cc:4841] TS 7c38a9fe2c584e36a383e6890a0f65b5 (127.2.164.65:35477): tablet 04d279549e8242809df92a242a31d97d (table test-table [id=b6a4438407964f8b8a3e0903edf0b933]) successfully deleted
I20260926 19:42:19.239656 17045 ts_tablet_manager.cc:1905] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:42:19.241631 17046 ts_tablet_manager.cc:1905] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:42:19.248028 17045 ts_tablet_manager.cc:1918] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:42:19.248225 17045 log.cc:1196] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/04d279549e8242809df92a242a31d97d
I20260926 19:42:19.248744 17045 ts_tablet_manager.cc:1939] T 04d279549e8242809df92a242a31d97d P 09f6d961f8e6451aac31f6c4a6ec16bd: Deleting consensus metadata
I20260926 19:42:19.250447 16533 catalog_manager.cc:4841] TS 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783): tablet 04d279549e8242809df92a242a31d97d (table test-table [id=b6a4438407964f8b8a3e0903edf0b933]) successfully deleted
Sep 26 19:42:19 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:42:19.254120 17046 ts_tablet_manager.cc:1918] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:42:19.254551 17046 log.cc:1196] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/04d279549e8242809df92a242a31d97d
I20260926 19:42:19.255152 17046 ts_tablet_manager.cc:1939] T 04d279549e8242809df92a242a31d97d P 9c78e3a48b444dddbc7b7074a0a811d2: Deleting consensus metadata
I20260926 19:42:19.257140 16531 catalog_manager.cc:4841] TS 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953): tablet 04d279549e8242809df92a242a31d97d (table test-table [id=b6a4438407964f8b8a3e0903edf0b933]) successfully deleted
I20260926 19:42:19.259570 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:19.221703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.2.164.65:35477)
0926 19:42:19.221909 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:19.221916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:19.227896 (+ 5980us) server_negotiation.cc:407] Connection header received
0926 19:42:19.227970 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:19.227977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:19.228061 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:19.228164 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:19.239106 (+ 10942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.240254 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.242287 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.242887 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.242982 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:19.252444 (+ 9462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:19.252461 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:19.252478 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:19.252521 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:19.256643 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:19.257608 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:19.257617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:19.257621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:19.257774 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:19.258200 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:19.258204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:19.258207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:19.258575 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:19.258791 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:19.259155 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 19:42:19.259312 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:42:19.260324 16731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:19.365304 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 16608
I20260926 19:42:19.380180 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 16742
Sep 26 19:42:19 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:42:19.398629 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:19.381724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.2.164.66:35953)
0926 19:42:19.381880 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:19.381885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:19.381899 (+ 14us) server_negotiation.cc:407] Connection header received
0926 19:42:19.381951 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:19.381954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:19.382024 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:19.382098 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:19.383689 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.384508 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.386549 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.387076 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.387180 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:19.392107 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:19.392120 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:19.392139 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:19.392195 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:19.395940 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:19.396593 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:19.396602 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:19.396606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:19.396764 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:19.397081 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:19.397086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:19.397089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:19.397530 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:19.397846 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:19.398013 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 19:42:19.398232 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 19:42:19.399308 16865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:19.512231 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 16742
I20260926 19:42:19.527042 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 16875
Sep 26 19:42:19 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:42:19.540728 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:19.527764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.2.164.67:34783)
0926 19:42:19.527915 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:19.527920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:19.527974 (+ 54us) server_negotiation.cc:407] Connection header received
0926 19:42:19.528116 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:19.528120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:19.528182 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:19.528274 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:19.529694 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.530430 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.531652 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:19.532027 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:19.532113 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:19.536062 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:19.536077 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:19.536093 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:19.536129 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:19.538558 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:19.539143 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:19.539149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:19.539152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:19.539287 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:19.539573 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:19.539576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:19.539578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:19.539909 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:19.540110 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:19.540339 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 19:42:19.540489 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:42:19.541448 16998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:19.630687 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 16875
I20260926 19:42:19.643920 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 16515
I20260926 19:42:19.645131 16577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:19.751988 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 16515
I20260926 19:42:19.768925 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36815
--webserver_interface=127.2.164.126
--webserver_port=37967
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:36815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:20.074831 17053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:20.075105 17053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:20.075239 17053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:20.082235 17053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:42:20.082352 17053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:20.082436 17053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:20.082511 17053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:42:20.082571 17053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:42:20.092047 17053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:36815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=37967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:20.093278 17053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:20.094491 17053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:20.117524 17059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:20.117466 17060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:20.117498 17062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:20.118162 17053 server_base.cc:1034] running on GCE node
I20260926 19:42:20.118944 17053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:20.120472 17053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:20.121672 17053 hybrid_clock.cc:648] HybridClock initialized: now 1790451740121612 us; error 54 us; skew 500 ppm
I20260926 19:42:20.122052 17053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:20 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451740, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:20.125341 17053 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:42:20.126892 17053 webserver.cc:460] Webserver started at http://127.2.164.126:37967/ using document root <none> and password file <none>
I20260926 19:42:20.127573 17053 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:20.127713 17053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:20.133488 17053 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 19:42:20.136356 17069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:20.137718 17053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:42:20.137894 17053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "85c29cdd8db244718bc1f1ea9abc2d49"
format_stamp: "Formatted at 2026-09-26 19:42:17 on dist-test-slave-mrw4"
server_key: "36c3066f4bf156a568657f0a1d3a2905"
server_key_iv: "e9612eb45dacc68ceead685168432d4a"
server_key_version: "encryptionkey@0"
I20260926 19:42:20.138931 17053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:20.179873 17053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:20.183982 17053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:20.184232 17053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:20.201799 17053 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:36815
I20260926 19:42:20.201812 17120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:36815 every 8 connection(s)
I20260926 19:42:20.203265 17053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:42:20.208784 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 17053
I20260926 19:42:20.209784 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:35477
--local_ip_for_outbound_sockets=127.2.164.65
--tserver_master_addrs=127.2.164.126:36815
--webserver_port=38929
--webserver_interface=127.2.164.65
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:42:20.210820 17121 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:42:20.215227 17121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Bootstrap starting.
I20260926 19:42:20.234656 17121 log.cc:824] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:20.246544 17121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:42:20.247017 17121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Bootstrap complete.
I20260926 19:42:20.255659 17121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:20.256481 17121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c29cdd8db244718bc1f1ea9abc2d49, State: Initialized, Role: FOLLOWER
I20260926 19:42:20.257120 17121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:20.257373 17121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:42:20.257486 17121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:42:20.257640 17121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:42:20.261538 17121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:20.262017 17121 leader_election.cc:302] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c29cdd8db244718bc1f1ea9abc2d49; no voters:
I20260926 19:42:20.262576 17121 leader_election.cc:288] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 19:42:20.262712 17125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 19:42:20.263767 17125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 2 LEADER]: Becoming Leader. State: Replica: 85c29cdd8db244718bc1f1ea9abc2d49, State: Running, Role: LEADER
I20260926 19:42:20.264389 17125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:20.264707 17121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:42:20.266729 17127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85c29cdd8db244718bc1f1ea9abc2d49. Latest consensus state: current_term: 2 leader_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } } }
I20260926 19:42:20.267071 17127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:20.267585 17126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } } }
I20260926 19:42:20.267859 17126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:20.272536 17132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:42:20.280789 17132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:42:20.282225 17132 catalog_manager.cc:1259] Loaded cluster ID: c3cbdcfb34be4b69b71a895bd9253179
I20260926 19:42:20.282408 17132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:42:20.285591 17132 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 19:42:20.302264 17132 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 19:42:20.309301 17132 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:42:20.310504 17132 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Loaded TSK: 0
I20260926 19:42:20.311539 17132 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 19:42:20.556169 17123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:20.556447 17123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:20.556578 17123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:20.563622 17123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:20.563735 17123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:20.563879 17123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:42:20.573525 17123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:35477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=38929
--tserver_master_addrs=127.2.164.126:36815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:20.574731 17123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:20.575994 17123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:20.590278 17149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:20.590255 17148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:20.590992 17151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:20.591295 17123 server_base.cc:1034] running on GCE node
I20260926 19:42:20.591885 17123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:20.592805 17123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:20.594002 17123 hybrid_clock.cc:648] HybridClock initialized: now 1790451740593936 us; error 62 us; skew 500 ppm
I20260926 19:42:20.594297 17123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:20 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451740, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:20.597277 17123 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:42:20.598707 17123 webserver.cc:460] Webserver started at http://127.2.164.65:38929/ using document root <none> and password file <none>
I20260926 19:42:20.599380 17123 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:20.599493 17123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:20.604563 17123 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:42:20.607190 17158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:20.608510 17123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:42:20.608641 17123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "7c38a9fe2c584e36a383e6890a0f65b5"
format_stamp: "Formatted at 2026-09-26 19:42:17 on dist-test-slave-mrw4"
server_key: "67ffe3b2bac69e76772dcee1c33edabf"
server_key_iv: "357bb82541eed9f83161601d9c0b2631"
server_key_version: "encryptionkey@0"
I20260926 19:42:20.609606 17123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:20.632942 17123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:20.637640 17123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:20.637897 17123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:20.638887 17123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:20.640321 17123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:20.640421 17123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:20.640515 17123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:20.640581 17123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:20.668358 17123 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:35477
I20260926 19:42:20.668423 17270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:35477 every 8 connection(s)
I20260926 19:42:20.669720 17123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:42:20.673462 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 17123
I20260926 19:42:20.674211 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:35953
--local_ip_for_outbound_sockets=127.2.164.66
--tserver_master_addrs=127.2.164.126:36815
--webserver_port=39735
--webserver_interface=127.2.164.66
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:42:20 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451740, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:20.692890 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:20.672466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:45811 (local address 127.2.164.126:36815)
0926 19:42:20.673097 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:20.673114 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:20.673839 (+ 725us) server_negotiation.cc:407] Connection header received
0926 19:42:20.676480 (+ 2641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:20.676493 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:20.676827 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:20.677122 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:20.678998 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:20.680270 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:20.681972 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:20.682604 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:20.682782 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:20.686143 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:20.686150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:20.686163 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:20.686201 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:20.688836 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:20.689719 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:20.689725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:20.689730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:20.689881 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:20.690356 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:20.690359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:20.690361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:20.690756 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:20.690899 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:20.691740 (+ 841us) server_negotiation.cc:299] Negotiation successful
0926 19:42:20.692028 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":226,"threads_started":1}
I20260926 19:42:20.693348 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:20.672779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:45811)
0926 19:42:20.673663 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:20.673712 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:20.675940 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:20.677342 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:20.677352 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:20.678016 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:20.678731 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:20.678756 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:20.680427 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:20.680432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:20.681839 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:20.681851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:20.682832 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:20.682839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:20.683080 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:20.683689 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:20.683713 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:20.685896 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:20.688978 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:20.688988 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:20.689002 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:20.689596 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:20.689979 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:20.689983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:20.689994 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:20.690248 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:20.690878 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:20.690885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:20.691284 (+ 399us) client_negotiation.cc:769] Sending connection context
0926 19:42:20.691586 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 19:42:20.692083 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":321,"threads_started":1}
I20260926 19:42:20.695930 17271 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36815
I20260926 19:42:20.696189 17271 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:20.696843 17271 heartbeater.cc:511] Master 127.2.164.126:36815 requested a full tablet report, sending...
I20260926 19:42:20.698925 17086 ts_manager.cc:194] Registered new tserver with Master: 7c38a9fe2c584e36a383e6890a0f65b5 (127.2.164.65:35477)
I20260926 19:42:20.700384 17086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:45811
W20260926 19:42:20.978163 17275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:20.978425 17275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:20.978557 17275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:20.985625 17275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:20.985735 17275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:20.985884 17275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:42:20.995116 17275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:35953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=39735
--tserver_master_addrs=127.2.164.126:36815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:20.996340 17275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:20.997658 17275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:21.010140 17281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:21.010167 17280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:21.010105 17283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:21.011178 17275 server_base.cc:1034] running on GCE node
I20260926 19:42:21.011708 17275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:21.012617 17275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:21.013809 17275 hybrid_clock.cc:648] HybridClock initialized: now 1790451741013778 us; error 39 us; skew 500 ppm
I20260926 19:42:21.014103 17275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:21 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:21.016923 17275 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:42:21.018261 17275 webserver.cc:460] Webserver started at http://127.2.164.66:39735/ using document root <none> and password file <none>
I20260926 19:42:21.018927 17275 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:21.019028 17275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:21.023893 17275 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 19:42:21.026326 17290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:21.027526 17275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:42:21.027657 17275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "9c78e3a48b444dddbc7b7074a0a811d2"
format_stamp: "Formatted at 2026-09-26 19:42:18 on dist-test-slave-mrw4"
server_key: "27d81f262e8ef9dc9a854023c930b0e2"
server_key_iv: "4831d34ba12abd5e5a95d8477358328f"
server_key_version: "encryptionkey@0"
I20260926 19:42:21.028556 17275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:21.043715 17275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:21.047605 17275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:21.047851 17275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:21.048871 17275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:21.050150 17275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:21.050241 17275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:21.050341 17275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:21.050410 17275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:21.079448 17275 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:35953
I20260926 19:42:21.079499 17402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:35953 every 8 connection(s)
I20260926 19:42:21.080824 17275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:42:21.085115 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 17275
I20260926 19:42:21.085832 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:34783
--local_ip_for_outbound_sockets=127.2.164.67
--tserver_master_addrs=127.2.164.126:36815
--webserver_port=38085
--webserver_interface=127.2.164.67
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:42:21 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:21.100857 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.083222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:60959 (local address 127.2.164.126:36815)
0926 19:42:21.083410 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:21.083414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:21.084512 (+ 1098us) server_negotiation.cc:407] Connection header received
0926 19:42:21.086869 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:21.086874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:21.086940 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:21.087024 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:21.088857 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.089585 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.091619 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.092026 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.092122 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.095350 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:21.095357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:21.095361 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:21.095401 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:21.098209 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:21.098810 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:21.098815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:21.098819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:21.098966 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:21.099294 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:21.099298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:21.099301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:21.099605 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:21.099749 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:21.100300 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 19:42:21.100527 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:42:21.101658 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.083698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:60959)
0926 19:42:21.084366 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:21.084403 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:21.086575 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:21.087273 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:21.087285 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:21.087939 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:21.088668 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.088691 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.089726 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.089731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.091470 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.091484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.092219 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.092225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.092474 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.093135 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:21.093156 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:21.095150 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:21.098334 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:21.098340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:21.098352 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:21.098710 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:21.099053 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:21.099056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:21.099059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:21.099211 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:21.099698 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:21.099704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:21.099983 (+ 279us) client_negotiation.cc:769] Sending connection context
0926 19:42:21.100182 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 19:42:21.100523 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":258,"threads_started":1}
I20260926 19:42:21.102933 17403 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36815
I20260926 19:42:21.103199 17403 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:21.103857 17403 heartbeater.cc:511] Master 127.2.164.126:36815 requested a full tablet report, sending...
I20260926 19:42:21.105137 17086 ts_manager.cc:194] Registered new tserver with Master: 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953)
I20260926 19:42:21.105945 17086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:60959
W20260926 19:42:21.392499 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:21.392760 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:21.392900 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:21.399832 17406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:21.399940 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:21.400085 17406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:42:21.409502 17406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:34783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=38085
--tserver_master_addrs=127.2.164.126:36815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:21.410666 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:21.411908 17406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:21.424281 17411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:21.424458 17412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:21.424458 17414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:21.425089 17406 server_base.cc:1034] running on GCE node
I20260926 19:42:21.425648 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:21.426520 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:21.427716 17406 hybrid_clock.cc:648] HybridClock initialized: now 1790451741427670 us; error 35 us; skew 500 ppm
I20260926 19:42:21.428020 17406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:21 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:21.431002 17406 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:42:21.432407 17406 webserver.cc:460] Webserver started at http://127.2.164.67:38085/ using document root <none> and password file <none>
I20260926 19:42:21.433068 17406 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:21.433218 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:21.438154 17406 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 19:42:21.440760 17421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:21.442135 17406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 19:42:21.442284 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "09f6d961f8e6451aac31f6c4a6ec16bd"
format_stamp: "Formatted at 2026-09-26 19:42:18 on dist-test-slave-mrw4"
server_key: "3551adf83dd4692e47805172033a97e6"
server_key_iv: "62863c39890006955542ab1b1ea5f168"
server_key_version: "encryptionkey@0"
I20260926 19:42:21.443244 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:21.456624 17406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:21.460420 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:21.460647 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:21.461522 17406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:21.462719 17406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:21.462821 17406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:21.462914 17406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:21.462980 17406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:21.491393 17406 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:34783
I20260926 19:42:21.491468 17533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:34783 every 8 connection(s)
I20260926 19:42:21.492703 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:42:21.498533 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 17406
Sep 26 19:42:21 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451741, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:21.512253 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.494996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:41213 (local address 127.2.164.126:36815)
0926 19:42:21.495174 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:21.495178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:21.496242 (+ 1064us) server_negotiation.cc:407] Connection header received
0926 19:42:21.497981 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:21.497986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:21.498061 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:21.498142 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:21.499818 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.500824 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.502325 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.502771 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.502889 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.506088 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:21.506092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:21.506096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:21.506126 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:21.509253 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:21.509922 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:21.509927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:21.509931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:21.510096 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:21.510529 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:21.510533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:21.510536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:21.510800 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:21.510950 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:21.511709 (+ 759us) server_negotiation.cc:299] Negotiation successful
0926 19:42:21.511933 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:42:21.512877 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.499426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.2.164.126:36815)
0926 19:42:21.500217 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:21.500223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:21.500241 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:42:21.500298 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:21.500302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:21.500372 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:21.500468 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:21.502030 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.502788 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.504116 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.504532 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.504638 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.507298 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:21.507304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:21.507309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:21.507361 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:21.510305 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:21.510914 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:21.510919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:21.510922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:21.511044 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:21.511418 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:21.511422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:21.511424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:21.511692 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:21.511933 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:21.512284 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 19:42:21.512492 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":566,"threads_started":1}
I20260926 19:42:21.513677 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.495458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:41213)
0926 19:42:21.496090 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:21.496130 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:21.497726 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:21.498371 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:21.498378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:21.498880 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:21.499617 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.499647 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.500962 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.500966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.502188 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.502204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.502983 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.502990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.503199 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.503779 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:21.503801 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:21.505892 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:21.509383 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:21.509390 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:21.509402 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:21.509794 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:21.510201 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:21.510205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:21.510207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:21.510393 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:21.510929 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:21.510935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:21.511284 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 19:42:21.511583 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 19:42:21.512078 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":240,"threads_started":1}
I20260926 19:42:21.514973 17534 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36815
I20260926 19:42:21.515264 17534 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:21.515959 17534 heartbeater.cc:511] Master 127.2.164.126:36815 requested a full tablet report, sending...
I20260926 19:42:21.517287 17085 ts_manager.cc:194] Registered new tserver with Master: 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783)
I20260926 19:42:21.518123 17085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:41213
I20260926 19:42:21.527701 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:42:21.534441 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.528732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.2.164.126:36815)
0926 19:42:21.528942 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:21.528948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:21.529271 (+ 323us) server_negotiation.cc:407] Connection header received
0926 19:42:21.529529 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:21.529533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:21.529659 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:21.529780 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:42:21.530190 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.530863 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.532351 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.532829 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.532932 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.533424 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:42:21.533906 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:42:21.534039 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:21.534124 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 19:42:21.534199 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 19:42:21.535274 17085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:42:21.542366 17085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:42:21.561995 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.553011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.0.0.1:55108)
0926 19:42:21.553978 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:21.554018 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:21.554186 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:21.555340 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:21.555347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:21.555377 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:21.555776 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.555795 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.557785 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.557789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.559898 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.559913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.561134 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.561140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.561330 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.561384 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 19:42:21.561517 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 19:42:21.561677 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":2489,"threads_started":1}
I20260926 19:42:21.562057 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.553263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.2.164.65:35477)
0926 19:42:21.554721 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:21.554731 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:21.554758 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:42:21.554854 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:21.554863 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:21.555125 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:21.555999 (+ 874us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:21.556221 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.557658 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.560040 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.560939 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.561075 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.561296 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:21.561563 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 19:42:21.561728 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1313,"thread_start_us":1215,"threads_started":1}
I20260926 19:42:21.563171 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.553223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:35953 (local address 127.0.0.1:49858)
0926 19:42:21.555022 (+ 1799us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:21.555037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:21.555134 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:21.556621 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:21.556627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:21.556652 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:21.556987 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.556997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.558604 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.558609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.560830 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.560843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.562558 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.562564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.562732 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.562762 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 19:42:21.562826 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 19:42:21.562924 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1696,"mutex_wait_us":4,"thread_start_us":1584,"threads_started":1}
I20260926 19:42:21.564208 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.554201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.2.164.66:35953)
0926 19:42:21.556043 (+ 1842us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:21.556050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:21.556081 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:42:21.556168 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:21.556174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:21.556401 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:21.557159 (+ 758us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:21.557418 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.558449 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.560981 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.562326 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.563468 (+ 1142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.563661 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:21.563721 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 19:42:21.563871 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1694,"thread_start_us":1564,"threads_started":1}
I20260926 19:42:21.564363 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.553719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:34783 (local address 127.0.0.1:43234)
0926 19:42:21.556697 (+ 2978us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:21.556714 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:21.556838 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:21.557317 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:21.557321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:21.557346 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:21.557653 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.557662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.559499 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.559504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.561521 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.561533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.563638 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.563645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.563812 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.563842 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 19:42:21.563926 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 19:42:21.564042 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2867,"mutex_wait_us":41,"thread_start_us":2745,"threads_started":1}
I20260926 19:42:21.564446 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.554814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.2.164.67:34783)
0926 19:42:21.555643 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:21.555651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:21.556805 (+ 1154us) server_negotiation.cc:407] Connection header received
0926 19:42:21.556977 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:21.556982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:21.557129 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:21.557316 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:21.557844 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.559346 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.561671 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.563438 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.563563 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.563767 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:21.563994 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 19:42:21.564133 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":519,"threads_started":1}
I20260926 19:42:21.566131 17206 tablet_service.cc:1467] Processing CreateTablet for tablet 14651b5384e74eabb75e9974e79d4927 (DEFAULT_TABLE table=test-table [id=c75b2bfb10b84f7b9105f2ca4526cef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:21.567431 17206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14651b5384e74eabb75e9974e79d4927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:21.567817 17338 tablet_service.cc:1467] Processing CreateTablet for tablet 14651b5384e74eabb75e9974e79d4927 (DEFAULT_TABLE table=test-table [id=c75b2bfb10b84f7b9105f2ca4526cef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:21.567826 17469 tablet_service.cc:1467] Processing CreateTablet for tablet 14651b5384e74eabb75e9974e79d4927 (DEFAULT_TABLE table=test-table [id=c75b2bfb10b84f7b9105f2ca4526cef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:21.569131 17469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14651b5384e74eabb75e9974e79d4927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:21.569136 17338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14651b5384e74eabb75e9974e79d4927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:21.578843 17550 tablet_bootstrap.cc:492] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Bootstrap starting.
I20260926 19:42:21.581766 17550 tablet_bootstrap.cc:654] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:21.582756 17550 log.cc:824] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:21.583616 17551 tablet_bootstrap.cc:492] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: Bootstrap starting.
I20260926 19:42:21.584347 17552 tablet_bootstrap.cc:492] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: Bootstrap starting.
I20260926 19:42:21.587523 17551 tablet_bootstrap.cc:654] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:21.587837 17552 tablet_bootstrap.cc:654] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:21.588665 17551 log.cc:824] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:21.588838 17552 log.cc:824] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:21.590114 17550 tablet_bootstrap.cc:492] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: No bootstrap required, opened a new log
I20260926 19:42:21.590377 17550 ts_tablet_manager.cc:1397] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20260926 19:42:21.591252 17552 tablet_bootstrap.cc:492] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: No bootstrap required, opened a new log
I20260926 19:42:21.591473 17551 tablet_bootstrap.cc:492] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: No bootstrap required, opened a new log
I20260926 19:42:21.591481 17552 ts_tablet_manager.cc:1397] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 19:42:21.591712 17551 ts_tablet_manager.cc:1397] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 19:42:21.598313 17552 raft_consensus.cc:348] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:21.598644 17552 raft_consensus.cc:374] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:21.598750 17552 raft_consensus.cc:729] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c78e3a48b444dddbc7b7074a0a811d2, State: Initialized, Role: FOLLOWER
I20260926 19:42:21.599326 17552 consensus_queue.cc:260] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:21.600366 17403 heartbeater.cc:503] Master 127.2.164.126:36815 was elected leader, sending a full tablet report...
I20260926 19:42:21.600946 17552 ts_tablet_manager.cc:1428] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20260926 19:42:21.600682 17550 raft_consensus.cc:348] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:21.601076 17550 raft_consensus.cc:374] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:21.601205 17550 raft_consensus.cc:729] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c38a9fe2c584e36a383e6890a0f65b5, State: Initialized, Role: FOLLOWER
I20260926 19:42:21.601428 17551 raft_consensus.cc:348] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:21.601807 17551 raft_consensus.cc:374] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:21.601902 17551 raft_consensus.cc:729] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f6d961f8e6451aac31f6c4a6ec16bd, State: Initialized, Role: FOLLOWER
I20260926 19:42:21.601807 17550 consensus_queue.cc:260] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:21.602491 17551 consensus_queue.cc:260] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:21.608865 17534 heartbeater.cc:503] Master 127.2.164.126:36815 was elected leader, sending a full tablet report...
I20260926 19:42:21.609156 17271 heartbeater.cc:503] Master 127.2.164.126:36815 was elected leader, sending a full tablet report...
I20260926 19:42:21.609625 17551 ts_tablet_manager.cc:1428] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: Time spent starting tablet: real 0.018s user 0.015s sys 0.005s
I20260926 19:42:21.610059 17550 ts_tablet_manager.cc:1428] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
W20260926 19:42:21.671721 17272 tablet.cc:2369] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:42:21.744218 17535 tablet.cc:2369] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:42:21.832654 17404 tablet.cc:2369] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:42:21.854357 17558 raft_consensus.cc:482] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:42:21.854632 17558 raft_consensus.cc:504] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:21.856249 17558 leader_election.cc:288] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c38a9fe2c584e36a383e6890a0f65b5 (127.2.164.65:35477), 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953)
I20260926 19:42:21.862267 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.856598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:38469 (local address 127.2.164.65:35477)
0926 19:42:21.856812 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:21.856820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:21.856843 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:42:21.857204 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:21.857210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:21.857289 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:21.857399 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:21.857755 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.858680 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.860438 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.861489 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.861597 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.861648 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:21.861990 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 19:42:21.862077 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:42:21.862366 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.856530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.2.164.67:38469)
0926 19:42:21.856727 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:21.856747 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:21.856893 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:21.857417 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:21.857421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:21.857445 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:21.857614 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.857626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.858810 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.858815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.860286 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.860297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.861625 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.861631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.861822 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.861857 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:42:21.861974 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 19:42:21.862107 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 19:42:21.863301 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.856694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:35953 (local address 127.2.164.67:49873)
0926 19:42:21.857473 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:21.857500 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:21.857611 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:21.857901 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:21.857905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:21.857928 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:21.858266 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.858276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.859463 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.859467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.861071 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:21.861085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.862733 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.862739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:21.862890 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.862930 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 19:42:21.863002 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 19:42:21.863084 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":401,"threads_started":1}
I20260926 19:42:21.863401 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.856933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:49873 (local address 127.2.164.66:35953)
0926 19:42:21.857255 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:21.857262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:21.857586 (+ 324us) server_negotiation.cc:407] Connection header received
0926 19:42:21.857725 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:21.857730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:21.857796 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:21.857864 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:21.858438 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.859302 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.861274 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.862342 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.862535 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.862617 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:21.863060 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 19:42:21.863186 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 19:42:21.863442 17226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14651b5384e74eabb75e9974e79d4927" candidate_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" is_pre_election: true
I20260926 19:42:21.864007 17226 raft_consensus.cc:2454] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f6d961f8e6451aac31f6c4a6ec16bd in term 0.
I20260926 19:42:21.864173 17358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14651b5384e74eabb75e9974e79d4927" candidate_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" is_pre_election: true
I20260926 19:42:21.864593 17358 raft_consensus.cc:2454] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f6d961f8e6451aac31f6c4a6ec16bd in term 0.
I20260926 19:42:21.864630 17425 leader_election.cc:302] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d961f8e6451aac31f6c4a6ec16bd, 7c38a9fe2c584e36a383e6890a0f65b5; no voters:
I20260926 19:42:21.865020 17558 raft_consensus.cc:2781] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:42:21.865227 17558 raft_consensus.cc:482] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:42:21.865342 17558 raft_consensus.cc:3037] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:21.867437 17558 raft_consensus.cc:504] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:21.867967 17558 leader_election.cc:288] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [CANDIDATE]: Term 1 election: Requested vote from peers 7c38a9fe2c584e36a383e6890a0f65b5 (127.2.164.65:35477), 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953)
I20260926 19:42:21.868357 17226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14651b5384e74eabb75e9974e79d4927" candidate_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c38a9fe2c584e36a383e6890a0f65b5"
I20260926 19:42:21.868469 17358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14651b5384e74eabb75e9974e79d4927" candidate_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c78e3a48b444dddbc7b7074a0a811d2"
I20260926 19:42:21.868525 17226 raft_consensus.cc:3037] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:21.868669 17358 raft_consensus.cc:3037] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:21.870702 17226 raft_consensus.cc:2454] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f6d961f8e6451aac31f6c4a6ec16bd in term 1.
I20260926 19:42:21.870894 17358 raft_consensus.cc:2454] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f6d961f8e6451aac31f6c4a6ec16bd in term 1.
I20260926 19:42:21.871054 17425 leader_election.cc:302] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f6d961f8e6451aac31f6c4a6ec16bd, 7c38a9fe2c584e36a383e6890a0f65b5; no voters:
I20260926 19:42:21.871335 17558 raft_consensus.cc:2781] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:42:21.871785 17558 raft_consensus.cc:686] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 1 LEADER]: Becoming Leader. State: Replica: 09f6d961f8e6451aac31f6c4a6ec16bd, State: Running, Role: LEADER
I20260926 19:42:21.872208 17558 consensus_queue.cc:237] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:21.876435 17086 catalog_manager.cc:5495] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67). New cstate: current_term: 1 leader_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } health_report { overall_health: HEALTHY } } }
I20260926 19:42:21.927358 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.922568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.2.164.67:34783)
0926 19:42:21.922727 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:21.922731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:21.922748 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:42:21.922834 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:21.922837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:21.922909 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:21.923026 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:42:21.923341 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.924061 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.925465 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.925850 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.925939 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.926374 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:42:21.926850 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:42:21.926953 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:21.927021 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 19:42:21.927083 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:42:21.935109 17226 raft_consensus.cc:1261] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Refusing update from remote peer 09f6d961f8e6451aac31f6c4a6ec16bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:42:21.935145 17358 raft_consensus.cc:1261] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2 [term 1 FOLLOWER]: Refusing update from remote peer 09f6d961f8e6451aac31f6c4a6ec16bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:42:21.935958 17558 consensus_queue.cc:1035] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:42:21.936223 17560 consensus_queue.cc:1035] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:42:21.946393 17563 mvcc.cc:205] Tried to move back new op lower bound from 7333690334956478464 to 7333690334712123392. Current Snapshot: MvccSnapshot[applied={T|T < 7333690334956478464}]
I20260926 19:42:21.955260 17564 mvcc.cc:205] Tried to move back new op lower bound from 7333690334956478464 to 7333690334712123392. Current Snapshot: MvccSnapshot[applied={T|T < 7333690334956478464 or (T in {7333690334956478464})}]
I20260926 19:42:21.960037 17085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:42:21.960342 17085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:42:21.963794 17085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Sending DeleteTablet for 3 replicas of tablet 14651b5384e74eabb75e9974e79d4927
I20260926 19:42:21.964974 17206 tablet_service.cc:1514] Processing DeleteTablet for tablet 14651b5384e74eabb75e9974e79d4927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:21 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:55108
I20260926 19:42:21.965003 17338 tablet_service.cc:1514] Processing DeleteTablet for tablet 14651b5384e74eabb75e9974e79d4927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:21 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:49858
I20260926 19:42:21.965241 17469 tablet_service.cc:1514] Processing DeleteTablet for tablet 14651b5384e74eabb75e9974e79d4927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:21 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:43234
I20260926 19:42:21.966404 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 17123
I20260926 19:42:21.984828 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:21.967263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.2.164.65:35477)
0926 19:42:21.967409 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:21.967415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:21.967522 (+ 107us) server_negotiation.cc:407] Connection header received
0926 19:42:21.967698 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:21.967705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:21.967784 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:21.967876 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:21.969613 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.970917 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.973087 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:21.973877 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:21.973989 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:21.977457 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:21.977477 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:21.977498 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:21.977542 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:21.981583 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:21.982449 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:21.982461 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:21.982465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:21.982656 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:21.983070 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:21.983075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:21.983079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:21.983670 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:21.983975 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:21.984282 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 19:42:21.984490 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 19:42:21.985739 17574 tablet_replica.cc:330] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: stopping tablet replica
I20260926 19:42:21.985978 17266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:21.986110 17573 tablet_replica.cc:330] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: stopping tablet replica
I20260926 19:42:21.986225 17574 raft_consensus.cc:2229] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:42:21.986505 17573 raft_consensus.cc:2229] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:42:21.986878 17574 raft_consensus.cc:2258] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:42:21.987041 17573 raft_consensus.cc:2258] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:42:21.989148 17573 ts_tablet_manager.cc:1905] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:42:21.989149 17574 ts_tablet_manager.cc:1905] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:42:21.996555 17574 ts_tablet_manager.cc:1918] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:42:21.996555 17573 ts_tablet_manager.cc:1918] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:42:21.996729 17573 log.cc:1196] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/14651b5384e74eabb75e9974e79d4927
I20260926 19:42:21.996727 17574 log.cc:1196] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/14651b5384e74eabb75e9974e79d4927
I20260926 19:42:21.997278 17574 ts_tablet_manager.cc:1939] T 14651b5384e74eabb75e9974e79d4927 P 09f6d961f8e6451aac31f6c4a6ec16bd: Deleting consensus metadata
I20260926 19:42:21.998157 17573 ts_tablet_manager.cc:1939] T 14651b5384e74eabb75e9974e79d4927 P 9c78e3a48b444dddbc7b7074a0a811d2: Deleting consensus metadata
I20260926 19:42:21.999280 17072 catalog_manager.cc:4841] TS 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783): tablet 14651b5384e74eabb75e9974e79d4927 (table test-table [id=c75b2bfb10b84f7b9105f2ca4526cef0]) successfully deleted
I20260926 19:42:21.999924 17070 catalog_manager.cc:4841] TS 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953): tablet 14651b5384e74eabb75e9974e79d4927 (table test-table [id=c75b2bfb10b84f7b9105f2ca4526cef0]) successfully deleted
I20260926 19:42:22.093713 17572 tablet_replica.cc:330] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: stopping tablet replica
I20260926 19:42:22.094190 17572 raft_consensus.cc:2229] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:42:22.094316 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 17123
W20260926 19:42:22.114140 17073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:35477 (error 108)
I20260926 19:42:22.114727 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 17275
I20260926 19:42:22.116108 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:22.115337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.0.0.1:55130)
0926 19:42:22.115804 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:22.115937 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:35477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":302,"threads_started":1}
W20260926 19:42:22.116430 17073 catalog_manager.cc:4584] TS 7c38a9fe2c584e36a383e6890a0f65b5 (127.2.164.65:35477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14651b5384e74eabb75e9974e79d4927: Network error: Client connection negotiation failed: client connection to 127.2.164.65:35477: connect: Connection refused (error 111)
I20260926 19:42:22.129127 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:22.115468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49860 (local address 127.2.164.66:35953)
0926 19:42:22.115632 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:22.115639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:22.115803 (+ 164us) server_negotiation.cc:407] Connection header received
0926 19:42:22.116006 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:22.116012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:22.116102 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:22.116196 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:22.117735 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:22.118803 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:22.119975 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:22.120601 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:22.120729 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:22.123040 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:22.123056 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:22.123075 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:22.123118 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:22.126520 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:22.127148 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:22.127157 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:22.127161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:22.127324 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:22.127632 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:22.127637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:22.127640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:22.128056 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:22.128350 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:22.128552 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 19:42:22.128755 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 19:42:22.129789 17398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:22.170249 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:22.169722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.0.0.1:55140)
0926 19:42:22.169954 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:22.170069 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:35477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 19:42:22.171484 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:22.171063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.0.0.1:55154)
0926 19:42:22.171226 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:22.171324 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:35477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 19:42:22.217303 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 17275
I20260926 19:42:22.232045 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:22.231502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.0.0.1:55166)
0926 19:42:22.231774 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:22.231859 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:35477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260926 19:42:22.232999 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:22.232681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.0.0.1:55182)
0926 19:42:22.232819 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:22.232882 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:35477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 19:42:22.237334 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 17406
I20260926 19:42:22.253666 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:22.238096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.2.164.67:34783)
0926 19:42:22.238282 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:22.238288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:22.238307 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:42:22.238468 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:22.238472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:22.238545 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:22.238630 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:22.239866 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:22.240716 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:22.241963 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:22.242438 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:22.242548 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:22.247867 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:22.247877 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:22.247893 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:22.247928 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:22.250825 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:22.251459 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:22.251466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:22.251469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:22.251614 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:22.251964 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:22.251967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:22.251970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:22.252320 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:22.252607 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:22.252858 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 19:42:22.253060 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:42:22.254320 17529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:22.313799 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:22.313270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.0.0.1:55186)
0926 19:42:22.313506 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:22.313618 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:35477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 19:42:22.314913 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:22.314467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.0.0.1:55202)
0926 19:42:22.314661 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:22.314749 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:35477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 19:42:22.372336 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 17406
I20260926 19:42:22.387723 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 17053
I20260926 19:42:22.388952 17116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:22.499575 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:22.498826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35477 (local address 127.0.0.1:55216)
0926 19:42:22.499106 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:22.499379 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:35477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 19:42:22.499802 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 17053
I20260926 19:42:22.517663 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36815
--webserver_interface=127.2.164.126
--webserver_port=37967
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:36815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:22.825434 17581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:22.825698 17581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:22.825793 17581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:22.832659 17581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:42:22.832749 17581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:22.832803 17581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:22.832856 17581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:42:22.832896 17581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:42:22.842310 17581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:36815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=37967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:22.843474 17581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:22.844695 17581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:22.855307 17586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:22.855792 17589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:22.855850 17587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:22.856149 17581 server_base.cc:1034] running on GCE node
I20260926 19:42:22.856861 17581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:22.858158 17581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:22.859359 17581 hybrid_clock.cc:648] HybridClock initialized: now 1790451742859306 us; error 43 us; skew 500 ppm
I20260926 19:42:22.859670 17581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:22 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451742, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:22.862533 17581 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:42:22.863912 17581 webserver.cc:460] Webserver started at http://127.2.164.126:37967/ using document root <none> and password file <none>
I20260926 19:42:22.864537 17581 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:22.864650 17581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:22.869760 17581 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:42:22.872467 17596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:22.873751 17581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:42:22.873907 17581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "85c29cdd8db244718bc1f1ea9abc2d49"
format_stamp: "Formatted at 2026-09-26 19:42:17 on dist-test-slave-mrw4"
server_key: "36c3066f4bf156a568657f0a1d3a2905"
server_key_iv: "e9612eb45dacc68ceead685168432d4a"
server_key_version: "encryptionkey@0"
I20260926 19:42:22.874805 17581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:22.894141 17581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:22.897928 17581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:22.898152 17581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:22.914602 17581 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:36815
I20260926 19:42:22.914621 17647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:36815 every 8 connection(s)
I20260926 19:42:22.915957 17581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:42:22.916865 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 17581
I20260926 19:42:22.917703 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:35477
--local_ip_for_outbound_sockets=127.2.164.65
--tserver_master_addrs=127.2.164.126:36815
--webserver_port=38929
--webserver_interface=127.2.164.65
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:42:22.923111 17648 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:42:22.926244 17648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Bootstrap starting.
I20260926 19:42:22.945580 17648 log.cc:824] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:22.961077 17648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:42:22.961588 17648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49: Bootstrap complete.
I20260926 19:42:22.970203 17648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:22.971114 17648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c29cdd8db244718bc1f1ea9abc2d49, State: Initialized, Role: FOLLOWER
I20260926 19:42:22.971714 17648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:22.971932 17648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:42:22.972033 17648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:42:22.972188 17648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 2 FOLLOWER]: Advancing to term 3
I20260926 19:42:22.975778 17648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:22.976222 17648 leader_election.cc:302] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c29cdd8db244718bc1f1ea9abc2d49; no voters:
I20260926 19:42:22.976801 17648 leader_election.cc:288] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 19:42:22.976930 17652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 19:42:22.977993 17652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [term 3 LEADER]: Becoming Leader. State: Replica: 85c29cdd8db244718bc1f1ea9abc2d49, State: Running, Role: LEADER
I20260926 19:42:22.978520 17652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } }
I20260926 19:42:22.978991 17648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:42:22.981107 17653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } } }
I20260926 19:42:22.981102 17654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85c29cdd8db244718bc1f1ea9abc2d49. Latest consensus state: current_term: 3 leader_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c29cdd8db244718bc1f1ea9abc2d49" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36815 } } }
I20260926 19:42:22.981519 17653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:22.981523 17654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c29cdd8db244718bc1f1ea9abc2d49 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:22.993867 17657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:42:23.002327 17657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:42:23.003762 17657 catalog_manager.cc:1259] Loaded cluster ID: c3cbdcfb34be4b69b71a895bd9253179
I20260926 19:42:23.003865 17657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 19:42:23.005852 17657 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd4337d29ad google::LogMessage::Fail() at ??:0
@ 0x7fd4337d588a google::LogMessage::SendToLog() at ??:0
@ 0x7fd4337d24de google::LogMessage::Flush() at ??:0
@ 0x7fd4337d3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd449205512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
I20260926 19:42:23.024942 17670 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fd449202976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd44928a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd44928a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd43641e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd43642191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd436421799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd4363fceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd4396d66db start_thread at ??:0
@ 0x7fd43266c61f clone at ??:0
W20260926 19:42:23.282213 17650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:23.282470 17650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:23.282598 17650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:23.289605 17650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:23.289703 17650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:23.289855 17650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:42:23.299116 17650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:35477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=38929
--tserver_master_addrs=127.2.164.126:36815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:23.300303 17650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:23.301783 17650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:23.314172 17689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:23.314134 17690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:23.314209 17692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:23.315008 17650 server_base.cc:1034] running on GCE node
I20260926 19:42:23.315578 17650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:23.316469 17650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:23.317687 17650 hybrid_clock.cc:648] HybridClock initialized: now 1790451743317639 us; error 38 us; skew 500 ppm
I20260926 19:42:23.317991 17650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:23 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451743, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:23.321027 17650 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:42:23.322398 17650 webserver.cc:460] Webserver started at http://127.2.164.65:38929/ using document root <none> and password file <none>
I20260926 19:42:23.323028 17650 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:23.323143 17650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:23.328168 17650 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 19:42:23.332756 17699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:23.334072 17650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:42:23.334209 17650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "7c38a9fe2c584e36a383e6890a0f65b5"
format_stamp: "Formatted at 2026-09-26 19:42:17 on dist-test-slave-mrw4"
server_key: "67ffe3b2bac69e76772dcee1c33edabf"
server_key_iv: "357bb82541eed9f83161601d9c0b2631"
server_key_version: "encryptionkey@0"
I20260926 19:42:23.335100 17650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:23.353842 17650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:23.357637 17650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:23.357867 17650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:23.358726 17650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:23.360391 17706 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 19:42:23.365579 17650 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 19:42:23.365697 17650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 19:42:23.365819 17650 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 19:42:23.368778 17650 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 19:42:23.368852 17650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260926 19:42:23.369273 17706 tablet_bootstrap.cc:492] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Bootstrap starting.
I20260926 19:42:23.395767 17706 log.cc:824] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:23.404019 17650 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:35477
I20260926 19:42:23.404057 17813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:35477 every 8 connection(s)
I20260926 19:42:23.405289 17650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:42:23.407722 17706 tablet_bootstrap.cc:492] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:42:23.408208 17706 tablet_bootstrap.cc:492] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Bootstrap complete.
I20260926 19:42:23.409222 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 17650
I20260926 19:42:23.409732 17706 ts_tablet_manager.cc:1397] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Time spent bootstrapping tablet: real 0.041s user 0.024s sys 0.015s
I20260926 19:42:23.409965 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:35953
--local_ip_for_outbound_sockets=127.2.164.66
--tserver_master_addrs=127.2.164.126:36815
--webserver_port=39735
--webserver_interface=127.2.164.66
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:42:23.410060 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:23.407948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:33255)
0926 19:42:23.408791 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:23.409090 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":367,"threads_started":1}
W20260926 19:42:23.410616 17814 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:36815 (0 consecutive failures): Network error: Failed to ping master at 127.2.164.126:36815: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
I20260926 19:42:23.411430 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:23.411100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:55523)
0926 19:42:23.411245 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:23.411317 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 19:42:23.412427 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:23.412070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:53103)
0926 19:42:23.412207 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:23.412292 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 19:42:23.412715 17814 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 19:42:23.418149 17706 raft_consensus.cc:348] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:23.418960 17706 raft_consensus.cc:729] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c38a9fe2c584e36a383e6890a0f65b5, State: Initialized, Role: FOLLOWER
I20260926 19:42:23.419507 17706 consensus_queue.cc:260] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:23.421005 17706 ts_tablet_manager.cc:1428] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260926 19:42:23.421061 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:23.420683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:40075)
0926 19:42:23.420839 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:23.420922 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 19:42:23.422384 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:23.421547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:44165)
0926 19:42:23.422062 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:23.422204 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":254,"threads_started":1}
W20260926 19:42:23.720468 17817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:23.720718 17817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:23.720835 17817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:23.727861 17817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:23.727952 17817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:23.728116 17817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:42:23.737392 17817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:35953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=39735
--tserver_master_addrs=127.2.164.126:36815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:23.738628 17817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:23.739980 17817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:23.752185 17825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:23.752378 17824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:23.752614 17827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:23.752941 17817 server_base.cc:1034] running on GCE node
I20260926 19:42:23.753548 17817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:23.754477 17817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:23.755677 17817 hybrid_clock.cc:648] HybridClock initialized: now 1790451743755627 us; error 43 us; skew 500 ppm
I20260926 19:42:23.755971 17817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:23 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451743, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:23.758934 17817 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:42:23.760229 17817 webserver.cc:460] Webserver started at http://127.2.164.66:39735/ using document root <none> and password file <none>
I20260926 19:42:23.760862 17817 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:23.760953 17817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:23.766136 17817 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:42:23.768775 17834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:23.770182 17817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:42:23.770320 17817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "9c78e3a48b444dddbc7b7074a0a811d2"
format_stamp: "Formatted at 2026-09-26 19:42:18 on dist-test-slave-mrw4"
server_key: "27d81f262e8ef9dc9a854023c930b0e2"
server_key_iv: "4831d34ba12abd5e5a95d8477358328f"
server_key_version: "encryptionkey@0"
I20260926 19:42:23.771258 17817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:23.798141 17817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:23.801887 17817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:23.802095 17817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:23.802958 17817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:23.804142 17817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:23.804224 17817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:23.804297 17817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:23.804347 17817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:23.833458 17817 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:35953
I20260926 19:42:23.833500 17946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:35953 every 8 connection(s)
I20260926 19:42:23.834815 17817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:42:23.839370 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:23.837626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:35449)
0926 19:42:23.838326 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:23.838575 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":253,"threads_started":1}
W20260926 19:42:23.840057 17947 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:36815 (0 consecutive failures): Network error: Failed to ping master at 127.2.164.126:36815: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
I20260926 19:42:23.840794 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:23.840499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:59207)
0926 19:42:23.840637 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:23.840701 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 19:42:23.841583 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:23.841284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:43447)
0926 19:42:23.841402 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:23.841475 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 19:42:23.841789 17947 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 19:42:23.842684 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 17817
I20260926 19:42:23.843251 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:34783
--local_ip_for_outbound_sockets=127.2.164.67
--tserver_master_addrs=127.2.164.126:36815
--webserver_port=38085
--webserver_interface=127.2.164.67
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:24.147534 17950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:24.147789 17950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:24.147929 17950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:24.154907 17950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:24.155015 17950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:24.155164 17950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:42:24.164280 17950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:34783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=38085
--tserver_master_addrs=127.2.164.126:36815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:24.165454 17950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:24.166687 17950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:24.178601 17958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:24.178686 17956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:24.179205 17955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:24.179211 17950 server_base.cc:1034] running on GCE node
I20260926 19:42:24.179859 17950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:24.180720 17950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:24.181910 17950 hybrid_clock.cc:648] HybridClock initialized: now 1790451744181845 us; error 51 us; skew 500 ppm
I20260926 19:42:24.182214 17950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:24 dist-test-slave-mrw4 krb5kdc[16502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451744, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:24.184988 17950 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:42:24.186367 17950 webserver.cc:460] Webserver started at http://127.2.164.67:38085/ using document root <none> and password file <none>
I20260926 19:42:24.186997 17950 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:24.187104 17950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:24.192195 17950 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 19:42:24.194828 17965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:24.196122 17950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:42:24.196269 17950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "09f6d961f8e6451aac31f6c4a6ec16bd"
format_stamp: "Formatted at 2026-09-26 19:42:18 on dist-test-slave-mrw4"
server_key: "3551adf83dd4692e47805172033a97e6"
server_key_iv: "62863c39890006955542ab1b1ea5f168"
server_key_version: "encryptionkey@0"
I20260926 19:42:24.197229 17950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:24.213105 17950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:24.216953 17950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:24.217217 17950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:24.218152 17950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:24.219411 17950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:24.219506 17950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:24.219609 17950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:24.219684 17950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:24.248987 17950 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:34783
I20260926 19:42:24.249033 18077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:34783 every 8 connection(s)
I20260926 19:42:24.250391 17950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:42:24.252583 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 17950
I20260926 19:42:24.255105 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:24.253224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:54215)
0926 19:42:24.253996 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:24.254241 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":301,"threads_started":1}
W20260926 19:42:24.255689 18078 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:36815 (0 consecutive failures): Network error: Failed to ping master at 127.2.164.126:36815: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
I20260926 19:42:24.256433 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:24.256107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:51579)
0926 19:42:24.256231 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:24.256326 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 19:42:24.257159 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:24.256880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:50733)
0926 19:42:24.256990 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:24.257058 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 19:42:24.257370 18078 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 19:42:24.423835 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:24.423122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:50559)
0926 19:42:24.423562 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:24.423641 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":226,"threads_started":1}
I20260926 19:42:24.843087 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:24.842323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:45561)
0926 19:42:24.842774 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:24.842864 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":206,"threads_started":1}
I20260926 19:42:24.894987 18084 raft_consensus.cc:482] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:42:24.895236 18084 raft_consensus.cc:504] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:24.896756 18084 leader_election.cc:288] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953), 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783)
Sep 26 19:42:24 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451743, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
Sep 26 19:42:24 dist-test-slave-mrw4 krb5kdc[16502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451743, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:42:24.919544 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:24.897469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:33843 (local address 127.2.164.67:34783)
0926 19:42:24.899960 (+ 2491us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:24.899975 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:24.900011 (+ 36us) server_negotiation.cc:407] Connection header received
0926 19:42:24.901582 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:24.901591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:24.901973 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:24.902253 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:24.903644 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:24.904764 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:24.906421 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:24.907031 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:24.907182 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:24.911261 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:24.911272 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:24.911290 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:24.911335 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:24.915008 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:24.916071 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:24.916080 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:24.916088 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:24.916285 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:24.916794 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:24.916799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:24.916802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:24.917538 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:24.917721 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:24.918720 (+ 999us) server_negotiation.cc:299] Negotiation successful
0926 19:42:24.919144 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2309,"thread_start_us":2184,"threads_started":1}
I20260926 19:42:24.919531 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:24.897324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:55353 (local address 127.2.164.66:35953)
0926 19:42:24.898801 (+ 1477us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:24.898819 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:24.898867 (+ 48us) server_negotiation.cc:407] Connection header received
0926 19:42:24.901105 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:24.901115 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:24.901562 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:24.901869 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:24.903649 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:24.904780 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:24.906415 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:24.907020 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:24.907182 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:24.911406 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:24.911416 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:24.911429 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:24.911470 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:24.915016 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:24.916071 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:24.916080 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:24.916088 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:24.916288 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:24.916794 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:24.916799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:24.916802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:24.917537 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:24.917721 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:24.918723 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0926 19:42:24.919137 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1270,"thread_start_us":1023,"threads_started":1}
I20260926 19:42:24.919545 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:24.897005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:35953 (local address 127.2.164.65:55353)
0926 19:42:24.897324 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:24.897369 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:24.897964 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:24.902063 (+ 4099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:24.902074 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:24.902731 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:24.903459 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:24.903485 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:24.904942 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:24.904946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:24.906286 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:24.906298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:24.907209 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:24.907215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:24.907525 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:24.908268 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:24.908285 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:24.911172 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:24.915179 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:24.915189 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:24.915206 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:24.915913 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:24.916397 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:24.916402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:24.916406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:24.916679 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:24.917686 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:24.917695 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:24.918156 (+ 461us) client_negotiation.cc:769] Sending connection context
0926 19:42:24.918474 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 19:42:24.919191 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":387,"spinlock_wait_cycles":30976}
I20260926 19:42:24.919555 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:24.897065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:34783 (local address 127.2.164.65:33843)
0926 19:42:24.899799 (+ 2734us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:24.899816 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:24.899924 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:24.902226 (+ 2302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:24.902231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:24.902826 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:24.903460 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:24.903485 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:24.904946 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:24.904949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:24.906285 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:24.906296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:24.907257 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:24.907265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:24.907520 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:24.908210 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:24.908231 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:24.910987 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:24.915179 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:24.915189 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:24.915206 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:24.915913 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:24.916422 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:24.916430 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:24.916433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:24.916679 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:24.917686 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:24.917695 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:24.918156 (+ 461us) client_negotiation.cc:769] Sending connection context
0926 19:42:24.918474 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 19:42:24.919053 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2629,"mutex_wait_us":199,"spinlock_wait_cycles":26496,"thread_start_us":2355,"threads_started":1}
W20260926 19:42:24.922940 17702 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
W20260926 19:42:24.923202 17700 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
I20260926 19:42:24.923347 17700 leader_election.cc:302] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c38a9fe2c584e36a383e6890a0f65b5; no voters: 09f6d961f8e6451aac31f6c4a6ec16bd, 9c78e3a48b444dddbc7b7074a0a811d2
I20260926 19:42:24.923739 18084 raft_consensus.cc:2726] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:42:25.258713 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:25.257863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:34469)
0926 19:42:25.258346 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:25.258492 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":248,"threads_started":1}
I20260926 19:42:25.425339 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:25.424671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:34133)
0926 19:42:25.425046 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:25.425123 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":182,"threads_started":1}
I20260926 19:42:25.844592 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:25.843959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:49461)
0926 19:42:25.844340 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:25.844429 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":198,"threads_started":1}
I20260926 19:42:26.260077 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:26.259461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:54355)
0926 19:42:26.259850 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:26.259928 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":204,"threads_started":1}
I20260926 19:42:26.426921 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:26.426130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:45621)
0926 19:42:26.426629 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:26.426721 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":216,"threads_started":1}
I20260926 19:42:26.846082 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:26.845431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:32965)
0926 19:42:26.845828 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:26.845905 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":201,"threads_started":1}
I20260926 19:42:26.857914 18094 raft_consensus.cc:482] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:42:26.858098 18094 raft_consensus.cc:504] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:26.858680 18094 leader_election.cc:288] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953), 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783)
W20260926 19:42:26.859607 17702 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
W20260926 19:42:26.859800 17700 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
I20260926 19:42:26.859925 17700 leader_election.cc:302] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c38a9fe2c584e36a383e6890a0f65b5; no voters: 09f6d961f8e6451aac31f6c4a6ec16bd, 9c78e3a48b444dddbc7b7074a0a811d2
I20260926 19:42:26.860177 18094 raft_consensus.cc:2726] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:42:27.261605 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:27.260876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:46657)
0926 19:42:27.261355 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:27.261436 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":222,"threads_started":1}
I20260926 19:42:27.428426 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:27.427748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:55509)
0926 19:42:27.428179 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:27.428257 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":235,"threads_started":1}
I20260926 19:42:27.847554 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:27.846866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:46171)
0926 19:42:27.847299 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:27.847381 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":217,"threads_started":1}
I20260926 19:42:28.263073 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:28.262392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:43607)
0926 19:42:28.262811 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:28.262894 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":209,"threads_started":1}
I20260926 19:42:28.430097 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:28.429416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:35817)
0926 19:42:28.429840 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:28.429917 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":210,"threads_started":1}
I20260926 19:42:28.849192 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:28.848476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:60583)
0926 19:42:28.848929 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:28.849004 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":251,"threads_started":1}
I20260926 19:42:29.264561 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:29.263897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:38173)
0926 19:42:29.264324 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:29.264402 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":223,"threads_started":1}
I20260926 19:42:29.431586 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:29.430899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:49195)
0926 19:42:29.431340 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:29.431425 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
I20260926 19:42:29.806416 18103 raft_consensus.cc:482] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:42:29.806622 18103 raft_consensus.cc:504] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:29.807304 18103 leader_election.cc:288] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953), 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783)
W20260926 19:42:29.808212 17700 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
W20260926 19:42:29.808401 17702 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
I20260926 19:42:29.808525 17702 leader_election.cc:302] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c38a9fe2c584e36a383e6890a0f65b5; no voters: 09f6d961f8e6451aac31f6c4a6ec16bd, 9c78e3a48b444dddbc7b7074a0a811d2
I20260926 19:42:29.808748 18103 raft_consensus.cc:2726] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:42:29.850682 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:29.849958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:46515)
0926 19:42:29.850428 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:29.850512 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":240,"threads_started":1}
I20260926 19:42:30.266081 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:30.265441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:43557)
0926 19:42:30.265838 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:30.265918 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":193,"threads_started":1}
I20260926 19:42:30.433003 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:30.432336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:52811)
0926 19:42:30.432736 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:30.432818 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":197,"threads_started":1}
I20260926 19:42:30.852223 18107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:30.851484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:57709)
0926 19:42:30.851960 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:30.852050 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":205,"threads_started":1}
I20260926 19:42:31.267731 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:31.266891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:54533)
0926 19:42:31.267442 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:31.267530 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":258,"threads_started":1}
I20260926 19:42:31.434554 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:31.433849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:40445)
0926 19:42:31.434301 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:31.434392 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":214,"threads_started":1}
I20260926 19:42:31.853760 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:31.853069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:46375)
0926 19:42:31.853512 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:31.853593 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":247,"threads_started":1}
I20260926 19:42:32.269337 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:32.268611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:37651)
0926 19:42:32.269006 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:32.269085 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":206,"threads_started":1}
I20260926 19:42:32.436067 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:32.435359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:38599)
0926 19:42:32.435798 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:32.435888 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":221,"threads_started":1}
I20260926 19:42:32.855316 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:32.854604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:52131)
0926 19:42:32.855062 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:32.855151 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":232,"threads_started":1}
I20260926 19:42:33.270857 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:33.270156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:42043)
0926 19:42:33.270624 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:33.270708 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":203,"threads_started":1}
I20260926 19:42:33.437644 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:33.436989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:33133)
0926 19:42:33.437404 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:33.437487 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":225,"threads_started":1}
I20260926 19:42:33.513288 18116 raft_consensus.cc:482] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:42:33.513520 18116 raft_consensus.cc:504] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:33.514308 18116 leader_election.cc:288] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953), 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783)
W20260926 19:42:33.515259 17702 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
W20260926 19:42:33.515604 17700 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
I20260926 19:42:33.515738 17700 leader_election.cc:302] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c38a9fe2c584e36a383e6890a0f65b5; no voters: 09f6d961f8e6451aac31f6c4a6ec16bd, 9c78e3a48b444dddbc7b7074a0a811d2
I20260926 19:42:33.515985 18116 raft_consensus.cc:2726] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:42:33.856889 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:33.856195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:60885)
0926 19:42:33.856622 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:33.856713 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":216,"threads_started":1}
I20260926 19:42:34.272469 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:34.271784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:40453)
0926 19:42:34.272209 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:34.272295 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":215,"threads_started":1}
I20260926 19:42:34.439431 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:34.438568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:53513)
0926 19:42:34.439105 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:34.439210 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":257,"threads_started":1}
I20260926 19:42:34.858662 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:34.857902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:40371)
0926 19:42:34.858402 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:34.858495 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":248,"threads_started":1}
I20260926 19:42:35.274101 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:35.273372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:59015)
0926 19:42:35.273832 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:35.273924 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":225,"threads_started":1}
I20260926 19:42:35.396472 18122 raft_consensus.cc:482] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:42:35.396648 18122 raft_consensus.cc:504] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:35.397368 18122 leader_election.cc:288] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953), 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783)
W20260926 19:42:35.398257 17702 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
W20260926 19:42:35.398411 17700 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
I20260926 19:42:35.398556 17700 leader_election.cc:302] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c38a9fe2c584e36a383e6890a0f65b5; no voters: 09f6d961f8e6451aac31f6c4a6ec16bd, 9c78e3a48b444dddbc7b7074a0a811d2
I20260926 19:42:35.398787 18122 raft_consensus.cc:2726] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:42:35.441202 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:35.440371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:34829)
0926 19:42:35.440892 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:35.440980 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":270,"threads_started":1}
I20260926 19:42:35.860167 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:35.859461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:39173)
0926 19:42:35.859925 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:35.860003 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":222,"threads_started":1}
I20260926 19:42:36.275689 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:36.274931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:43939)
0926 19:42:36.275416 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:36.275521 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
I20260926 19:42:36.442682 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:36.441993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:36719)
0926 19:42:36.442404 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:36.442485 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":200,"threads_started":1}
I20260926 19:42:36.861651 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:36.860956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:37649)
0926 19:42:36.861395 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:36.861492 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260926 19:42:37.100946 18128 raft_consensus.cc:482] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:42:37.101155 18128 raft_consensus.cc:504] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c38a9fe2c584e36a383e6890a0f65b5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35477 } } peers { permanent_uuid: "9c78e3a48b444dddbc7b7074a0a811d2" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 35953 } } peers { permanent_uuid: "09f6d961f8e6451aac31f6c4a6ec16bd" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 34783 } }
I20260926 19:42:37.101778 18128 leader_election.cc:288] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953), 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783)
W20260926 19:42:37.102805 17700 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c78e3a48b444dddbc7b7074a0a811d2 (127.2.164.66:35953): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
W20260926 19:42:37.102990 17702 leader_election.cc:341] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09f6d961f8e6451aac31f6c4a6ec16bd (127.2.164.67:34783): Not found: Tablet not found: 14651b5384e74eabb75e9974e79d4927
I20260926 19:42:37.103088 17702 leader_election.cc:302] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c38a9fe2c584e36a383e6890a0f65b5; no voters: 09f6d961f8e6451aac31f6c4a6ec16bd, 9c78e3a48b444dddbc7b7074a0a811d2
I20260926 19:42:37.103314 18128 raft_consensus.cc:2726] T 14651b5384e74eabb75e9974e79d4927 P 7c38a9fe2c584e36a383e6890a0f65b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:42:37.277297 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:37.276531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:50725)
0926 19:42:37.277004 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:37.277101 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260926 19:42:37.444272 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:37.443441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:51333)
0926 19:42:37.443970 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:37.444073 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":261,"threads_started":1}
I20260926 19:42:37.863183 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:37.862456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:55749)
0926 19:42:37.862891 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:37.862993 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":207,"threads_started":1}
I20260926 19:42:38.278863 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:38.278085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:59051)
0926 19:42:38.278542 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:38.278623 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":214,"threads_started":1}
I20260926 19:42:38.446053 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:38.445319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.65:39475)
0926 19:42:38.445773 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:38.445879 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":259,"threads_started":1}
I20260926 19:42:38.864935 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:38.864214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.66:53801)
0926 19:42:38.864675 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:38.864772 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
I20260926 19:42:39.257705 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 17650
I20260926 19:42:39.272953 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:39.258695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.2.164.65:35477)
0926 19:42:39.259294 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:39.259305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:39.259336 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:42:39.259423 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:39.259433 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:39.259689 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:39.259858 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:39.262004 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:39.263028 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:39.264320 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:39.264876 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:39.264974 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:39.267604 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:39.267616 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:39.267630 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:39.267665 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:39.270535 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:39.271153 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:39.271160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:39.271162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:39.271289 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:39.271619 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:39.271623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:39.271625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:39.272005 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:39.272253 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:39.272478 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 19:42:39.272677 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":247,"threads_started":1}
I20260926 19:42:39.274225 17809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:39.280707 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:39.279977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36815 (local address 127.2.164.67:52297)
0926 19:42:39.280438 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:39.280560 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:36815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":234,"threads_started":1}
I20260926 19:42:39.361824 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 17650
I20260926 19:42:39.376964 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 17817
I20260926 19:42:39.390259 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:39.377656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.2.164.66:35953)
0926 19:42:39.378113 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:39.378117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:39.378132 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:42:39.378187 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:39.378191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:39.378248 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:39.378338 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:39.379770 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:39.380618 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:39.381897 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:39.382320 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:39.382426 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:39.384900 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:39.384905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:39.384908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:39.384942 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:39.387410 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:39.388081 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:39.388087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:39.388090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:39.388221 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:39.388589 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:39.388593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:39.388595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:39.388875 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:39.389117 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:39.389678 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 19:42:39.389899 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260926 19:42:39.390919 17942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:39.477270 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 17817
I20260926 19:42:39.491465 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 17950
I20260926 19:42:39.504158 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:39.492264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.2.164.67:34783)
0926 19:42:39.492812 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:39.492817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:39.492834 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:42:39.492889 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:39.492894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:39.492960 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:39.493050 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:39.494396 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:39.495181 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:39.496355 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:39.496746 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:39.496831 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:39.499173 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:39.499177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:39.499180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:39.499211 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:39.501988 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:39.502606 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:39.502609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:39.502611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:39.502737 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:39.503001 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:39.503004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:39.503006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:39.503241 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:39.503449 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:39.503655 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 19:42:39.503811 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":279,"threads_started":1}
I20260926 19:42:39.504879 18073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:39.587579 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 17950
2026-09-26T19:42:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23983 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:42:39 dist-test-slave-mrw4 krb5kdc[18144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:42:39 dist-test-slave-mrw4 krb5kdc[18144](info): set up 2 sockets
Sep 26 19:42:39 dist-test-slave-mrw4 krb5kdc[18144](info): commencing operation
krb5kdc: starting...
W20260926 19:42:40.593569 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:42:40 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:42:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:42:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:42:40.797189 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44997
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:44997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:41.107155 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:41.107416 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:41.107549 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:41.114478 18157 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 19:42:41.114570 18157 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 19:42:41.114630 18157 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 19:42:41.114701 18157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:42:41.114802 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:41.114897 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:41.114959 18157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:42:41.115029 18157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:42:41.124400 18157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:44997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:41.125615 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:41.126845 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:41.137595 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:41.137637 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:41.137744 18162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:41.138554 18157 server_base.cc:1034] running on GCE node
I20260926 19:42:41.139222 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:41.140435 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:41.141625 18157 hybrid_clock.cc:648] HybridClock initialized: now 1790451761141561 us; error 38 us; skew 500 ppm
I20260926 19:42:41.141939 18157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:41 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451761, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:41.144788 18157 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:42:41.146158 18157 webserver.cc:460] Webserver started at http://127.2.164.126:35213/ using document root <none> and password file <none>
I20260926 19:42:41.146793 18157 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:41.146888 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:41.147161 18157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:41.149989 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "39ccbb9a19104808afd74ab8795ad074"
format_stamp: "Formatted at 2026-09-26 19:42:41 on dist-test-slave-mrw4"
server_key: "0428e91f327a5fb946b32f758d169012"
server_key_iv: "284a712c35936cdfe1bc3039ac6c52f1"
server_key_version: "encryptionkey@0"
I20260926 19:42:41.150703 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "39ccbb9a19104808afd74ab8795ad074"
format_stamp: "Formatted at 2026-09-26 19:42:41 on dist-test-slave-mrw4"
server_key: "0428e91f327a5fb946b32f758d169012"
server_key_iv: "284a712c35936cdfe1bc3039ac6c52f1"
server_key_version: "encryptionkey@0"
I20260926 19:42:41.155427 18157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:42:41.158591 18172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:41.159828 18157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 19:42:41.159967 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "39ccbb9a19104808afd74ab8795ad074"
format_stamp: "Formatted at 2026-09-26 19:42:41 on dist-test-slave-mrw4"
server_key: "0428e91f327a5fb946b32f758d169012"
server_key_iv: "284a712c35936cdfe1bc3039ac6c52f1"
server_key_version: "encryptionkey@0"
I20260926 19:42:41.160144 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:41.187093 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:41.190681 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:41.190912 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:41.208112 18157 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:44997
I20260926 19:42:41.208132 18223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:44997 every 8 connection(s)
I20260926 19:42:41.209445 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:42:41.213045 18224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:41.215382 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 18157
I20260926 19:42:41.215644 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:42:41.216420 2705 external_mini_cluster.cc:1428] Setting key 2e02c335185075936c99055fa73cba38
I20260926 19:42:41.225430 18224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Bootstrap starting.
I20260926 19:42:41.229063 18224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:41.230373 18224 log.cc:824] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:42:41 dist-test-slave-mrw4 krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:41.233403 18224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: No bootstrap required, opened a new log
I20260926 19:42:41.239362 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:41.218453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.2.164.126:44997)
0926 19:42:41.219213 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:41.219224 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:41.219269 (+ 45us) server_negotiation.cc:407] Connection header received
0926 19:42:41.220870 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:41.220891 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:41.221300 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:41.221622 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:41.223487 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:41.224828 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:41.226689 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:41.227385 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:41.227512 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:41.232391 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:41.232399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:41.232414 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:41.232448 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:41.235377 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:41.236251 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:41.236257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:41.236262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:41.236429 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:41.236789 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:41.236792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:41.236794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:41.237341 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:41.237549 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:41.237981 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 19:42:41.238340 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":354,"threads_started":1}
I20260926 19:42:41.242187 18224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:42:41.242600 18224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:41.242689 18224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ccbb9a19104808afd74ab8795ad074, State: Initialized, Role: FOLLOWER
I20260926 19:42:41.243149 18224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:42:41.243352 18224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:42:41.243444 18224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:42:41.243556 18224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:41.245573 18224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:42:41.246069 18224 leader_election.cc:302] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39ccbb9a19104808afd74ab8795ad074; no voters:
I20260926 19:42:41.246600 18224 leader_election.cc:288] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:42:41.246740 18229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:42:41.247390 18229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 1 LEADER]: Becoming Leader. State: Replica: 39ccbb9a19104808afd74ab8795ad074, State: Running, Role: LEADER
I20260926 19:42:41.247896 18229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:42:41.248507 18224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:42:41.250156 18230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39ccbb9a19104808afd74ab8795ad074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } } }
I20260926 19:42:41.250557 18230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:41.250475 18231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ccbb9a19104808afd74ab8795ad074. Latest consensus state: current_term: 1 leader_uuid: "39ccbb9a19104808afd74ab8795ad074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } } }
I20260926 19:42:41.250721 18231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: This master's current role is: LEADER
I20260926 19:42:41.252336 18235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:42:41.256310 18235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:42:41.264209 18235 catalog_manager.cc:1347] Generated new cluster ID: 2ba1a8674f0e4f9681123c93617e1fd6
I20260926 19:42:41.264302 18235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:42:41.280984 18235 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:42:41.282233 18235 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:42:41.295815 18235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Generated new TSK 0
I20260926 19:42:41.296627 18235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:42:41.352393 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44997
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:41.669600 18250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:41.669863 18250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:41.669987 18250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:41.676980 18250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:41.677066 18250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:41.677217 18250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:42:41.686360 18250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:41.687520 18250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:41.688755 18250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:41.700886 18256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:41.700955 18255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:41.700980 18258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:41.701526 18250 server_base.cc:1034] running on GCE node
I20260926 19:42:41.702078 18250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:41.702920 18250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:41.704114 18250 hybrid_clock.cc:648] HybridClock initialized: now 1790451761704075 us; error 27 us; skew 500 ppm
I20260926 19:42:41.704424 18250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:41 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451761, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:41.712162 18250 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:42:41.713620 18250 webserver.cc:460] Webserver started at http://127.2.164.65:44255/ using document root <none> and password file <none>
I20260926 19:42:41.714304 18250 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:41.714416 18250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:41.714769 18250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:41.717751 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "6f154833b09c4e179629745257d9a417"
format_stamp: "Formatted at 2026-09-26 19:42:41 on dist-test-slave-mrw4"
server_key: "7cb42e318407f265ca07f9ba3775a532"
server_key_iv: "781edd97fee3abfb4d3c84e51d52c020"
server_key_version: "encryptionkey@0"
I20260926 19:42:41.718469 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "6f154833b09c4e179629745257d9a417"
format_stamp: "Formatted at 2026-09-26 19:42:41 on dist-test-slave-mrw4"
server_key: "7cb42e318407f265ca07f9ba3775a532"
server_key_iv: "781edd97fee3abfb4d3c84e51d52c020"
server_key_version: "encryptionkey@0"
I20260926 19:42:41.723424 18250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 19:42:41.726756 18265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:41.728114 18250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:42:41.728255 18250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "6f154833b09c4e179629745257d9a417"
format_stamp: "Formatted at 2026-09-26 19:42:41 on dist-test-slave-mrw4"
server_key: "7cb42e318407f265ca07f9ba3775a532"
server_key_iv: "781edd97fee3abfb4d3c84e51d52c020"
server_key_version: "encryptionkey@0"
I20260926 19:42:41.728421 18250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:41.753602 18250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:41.757344 18250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:41.757584 18250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:41.758491 18250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:41.759719 18250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:41.759788 18250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:41.759879 18250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:41.759932 18250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:41.789055 18250 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:37793
I20260926 19:42:41.789098 18377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:37793 every 8 connection(s)
I20260926 19:42:41.790319 18250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:42:41.796832 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 18250
I20260926 19:42:41.797037 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:42:41.797837 2705 external_mini_cluster.cc:1428] Setting key 569e041bae2dd84fe02dd3901d5f8f18
Sep 26 19:42:41 dist-test-slave-mrw4 krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451761, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:41.810330 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:41.792665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:38861 (local address 127.2.164.126:44997)
0926 19:42:41.793078 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:41.793083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:41.793958 (+ 875us) server_negotiation.cc:407] Connection header received
0926 19:42:41.795732 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:41.795738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:41.795815 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:41.795926 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:41.797902 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:41.798747 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:41.800621 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:41.801200 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:41.801330 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:41.804573 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:41.804593 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:41.804597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:41.804637 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:41.807417 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:41.808220 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:41.808224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:41.808226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:41.808338 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:41.808765 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:41.808768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:41.808770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:41.809015 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:41.809123 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:41.809907 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 19:42:41.810073 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":213,"threads_started":1}
I20260926 19:42:41.811542 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:41.793093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:38861)
0926 19:42:41.793804 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:41.793841 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:41.795472 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:41.796175 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:41.796183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:41.796855 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:41.797665 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:41.797717 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:41.798909 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:41.798914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:41.800433 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:41.800444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:41.801383 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:41.801388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:41.801576 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:41.802186 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:41.802206 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:41.804361 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:41.807544 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:41.807553 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:41.807580 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:41.808111 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:41.808438 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:41.808443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:41.808447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:41.808676 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:41.809123 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:41.809130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:41.809520 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 19:42:41.809794 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 19:42:41.810265 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":258,"threads_started":1}
I20260926 19:42:41.812999 18378 heartbeater.cc:347] Connected to a master server at 127.2.164.126:44997
I20260926 19:42:41.813336 18378 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:41.814013 18378 heartbeater.cc:511] Master 127.2.164.126:44997 requested a full tablet report, sending...
I20260926 19:42:41.815958 18189 ts_manager.cc:194] Registered new tserver with Master: 6f154833b09c4e179629745257d9a417 (127.2.164.65:37793)
I20260926 19:42:41.817405 18189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:38861
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:42:41.848270 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44997
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:42.155827 18384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:42.156085 18384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:42.156186 18384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:42.163314 18384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:42.163410 18384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:42.163527 18384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:42:42.172991 18384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:42.174201 18384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:42.175426 18384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:42.187736 18389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:42.187716 18390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:42.187716 18392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:42.189025 18384 server_base.cc:1034] running on GCE node
I20260926 19:42:42.189669 18384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:42.190575 18384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:42.191792 18384 hybrid_clock.cc:648] HybridClock initialized: now 1790451762191743 us; error 34 us; skew 500 ppm
I20260926 19:42:42.192118 18384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:42 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451762, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:42.195276 18384 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:42:42.196699 18384 webserver.cc:460] Webserver started at http://127.2.164.66:36599/ using document root <none> and password file <none>
I20260926 19:42:42.197408 18384 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:42.197511 18384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:42.197803 18384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:42.200630 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "894784a0522c4e008d2b2114bde2cd6f"
format_stamp: "Formatted at 2026-09-26 19:42:42 on dist-test-slave-mrw4"
server_key: "427be354d4e2440cf154059259739baf"
server_key_iv: "3d2998f0e96d78a43b04137b12af08f2"
server_key_version: "encryptionkey@0"
I20260926 19:42:42.201376 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "894784a0522c4e008d2b2114bde2cd6f"
format_stamp: "Formatted at 2026-09-26 19:42:42 on dist-test-slave-mrw4"
server_key: "427be354d4e2440cf154059259739baf"
server_key_iv: "3d2998f0e96d78a43b04137b12af08f2"
server_key_version: "encryptionkey@0"
I20260926 19:42:42.206439 18384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:42:42.209977 18399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:42.211376 18384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:42:42.211527 18384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "894784a0522c4e008d2b2114bde2cd6f"
format_stamp: "Formatted at 2026-09-26 19:42:42 on dist-test-slave-mrw4"
server_key: "427be354d4e2440cf154059259739baf"
server_key_iv: "3d2998f0e96d78a43b04137b12af08f2"
server_key_version: "encryptionkey@0"
I20260926 19:42:42.211712 18384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:42.229840 18384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:42.233902 18384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:42.234155 18384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:42.235020 18384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:42.236271 18384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:42.236344 18384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:42.236466 18384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:42.236531 18384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:42.266059 18384 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:36773
I20260926 19:42:42.266114 18511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:36773 every 8 connection(s)
I20260926 19:42:42.267251 18384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:42:42.277496 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 18384
I20260926 19:42:42.277686 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:42:42.278194 2705 external_mini_cluster.cc:1428] Setting key 6851c97efec86e26db7e2fb87359b185
Sep 26 19:42:42 dist-test-slave-mrw4 krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451762, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:42.286197 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.269649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:56095 (local address 127.2.164.126:44997)
0926 19:42:42.269847 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:42.269854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:42.270951 (+ 1097us) server_negotiation.cc:407] Connection header received
0926 19:42:42.272632 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:42.272636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:42.272703 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:42.272787 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:42.274214 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.274919 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.276381 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.276822 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.276929 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.280762 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:42.280767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:42.280770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:42.280802 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:42.283339 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:42.284114 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:42.284117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:42.284119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:42.284243 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:42.284659 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:42.284662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:42.284664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:42.284915 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:42.285032 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:42.285768 (+ 736us) server_negotiation.cc:299] Negotiation successful
0926 19:42:42.285940 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:42:42.287424 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.270134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:56095)
0926 19:42:42.270800 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:42.270840 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:42.272396 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:42.272968 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:42.272976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:42.273526 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:42.274061 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.274078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.275034 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.275037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.276237 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.276248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.277100 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.277108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.277442 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.278157 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:42.278193 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:42.280540 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:42.283463 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:42.283473 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:42.283488 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:42.284006 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:42.284337 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:42.284341 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:42.284345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:42.284566 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:42.285020 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:42.285027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:42.285439 (+ 412us) client_negotiation.cc:769] Sending connection context
0926 19:42:42.285712 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 19:42:42.286179 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":257,"threads_started":1}
I20260926 19:42:42.288741 18512 heartbeater.cc:347] Connected to a master server at 127.2.164.126:44997
I20260926 19:42:42.289033 18512 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:42.289840 18512 heartbeater.cc:511] Master 127.2.164.126:44997 requested a full tablet report, sending...
I20260926 19:42:42.291344 18189 ts_manager.cc:194] Registered new tserver with Master: 894784a0522c4e008d2b2114bde2cd6f (127.2.164.66:36773)
I20260926 19:42:42.292122 18189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:56095
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:42:42.328050 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44997
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:42:42.636763 18517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:42:42.637012 18517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:42:42.637144 18517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:42:42.644106 18517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:42:42.644205 18517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:42:42.644352 18517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:42:42.653669 18517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:44997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:42:42.654795 18517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:42:42.655997 18517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:42:42.667847 18525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:42.667896 18523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:42:42.667920 18522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:42:42.668919 18517 server_base.cc:1034] running on GCE node
I20260926 19:42:42.669466 18517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:42:42.670251 18517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:42:42.671434 18517 hybrid_clock.cc:648] HybridClock initialized: now 1790451762671397 us; error 31 us; skew 500 ppm
I20260926 19:42:42.671689 18517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:42:42 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451762, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:42:42.674461 18517 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:42:42.675771 18517 webserver.cc:460] Webserver started at http://127.2.164.67:43145/ using document root <none> and password file <none>
I20260926 19:42:42.676384 18517 fs_manager.cc:359] Metadata directory not provided
I20260926 19:42:42.676488 18517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:42:42.676775 18517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:42:42.679543 18517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "258f0ada9c8b4290b285e0a4fc438032"
format_stamp: "Formatted at 2026-09-26 19:42:42 on dist-test-slave-mrw4"
server_key: "9ece45e3717b8219a0904072990aeee3"
server_key_iv: "773be7891651db28076ee0c9af041d6c"
server_key_version: "encryptionkey@0"
I20260926 19:42:42.680239 18517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "258f0ada9c8b4290b285e0a4fc438032"
format_stamp: "Formatted at 2026-09-26 19:42:42 on dist-test-slave-mrw4"
server_key: "9ece45e3717b8219a0904072990aeee3"
server_key_iv: "773be7891651db28076ee0c9af041d6c"
server_key_version: "encryptionkey@0"
I20260926 19:42:42.684911 18517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260926 19:42:42.688110 18532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:42.689373 18517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:42:42.689507 18517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "258f0ada9c8b4290b285e0a4fc438032"
format_stamp: "Formatted at 2026-09-26 19:42:42 on dist-test-slave-mrw4"
server_key: "9ece45e3717b8219a0904072990aeee3"
server_key_iv: "773be7891651db28076ee0c9af041d6c"
server_key_version: "encryptionkey@0"
I20260926 19:42:42.689693 18517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:42:42.705582 18517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:42:42.709499 18517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:42:42.709758 18517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:42:42.710697 18517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:42:42.711959 18517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:42:42.712034 18517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:42.712148 18517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:42:42.712203 18517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:42:42.742066 18517 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:43551
I20260926 19:42:42.742110 18644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:43551 every 8 connection(s)
I20260926 19:42:42.743283 18517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:42:42.747272 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 18517
I20260926 19:42:42.747459 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:42:42.747944 2705 external_mini_cluster.cc:1428] Setting key b4e46fc95b51a8338aba6a58b320c4c9
Sep 26 19:42:42 dist-test-slave-mrw4 krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451762, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:42:42.761879 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.745806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:34559 (local address 127.2.164.126:44997)
0926 19:42:42.745970 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:42.745975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:42.747131 (+ 1156us) server_negotiation.cc:407] Connection header received
0926 19:42:42.749185 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:42.749191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:42.749275 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:42.749349 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:42.751123 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.751863 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.753348 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.753745 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.753833 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.756916 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:42.756920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:42.756923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:42.756955 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:42.759269 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:42.759892 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:42.759895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:42.759897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:42.760014 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:42.760335 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:42.760338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:42.760339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:42.760576 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:42.760680 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:42.761309 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 19:42:42.761546 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:42:42.762748 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.746209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:34559)
0926 19:42:42.746950 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:42.746992 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:42.748890 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:42.749637 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:42.749646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:42.750249 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:42:42.750961 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.750985 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.751993 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.751996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.753240 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.753252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.753921 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.753926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.754123 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.754664 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:42:42.754684 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:42:42.756740 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:42:42.759384 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:42.759392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:42.759404 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:42.759793 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:42.760093 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:42:42.760097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:42:42.760099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:42:42.760260 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:42:42.760673 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:42:42.760679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:42:42.760959 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 19:42:42.761203 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 19:42:42.761603 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":318,"threads_started":1}
I20260926 19:42:42.763965 18645 heartbeater.cc:347] Connected to a master server at 127.2.164.126:44997
I20260926 19:42:42.764256 18645 heartbeater.cc:464] Registering TS with master...
I20260926 19:42:42.764972 18645 heartbeater.cc:511] Master 127.2.164.126:44997 requested a full tablet report, sending...
I20260926 19:42:42.766335 18189 ts_manager.cc:194] Registered new tserver with Master: 258f0ada9c8b4290b285e0a4fc438032 (127.2.164.67:43551)
I20260926 19:42:42.767072 18189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:34559
I20260926 19:42:42.775205 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:42:42.791033 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.779419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.2.164.126:44997)
0926 19:42:42.779595 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:42.779599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:42.779960 (+ 361us) server_negotiation.cc:407] Connection header received
0926 19:42:42.780122 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:42.780126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:42.780183 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:42.780262 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:42.781703 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.782382 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.783713 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.784101 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.784201 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.786348 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:42.786353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:42.786356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:42.786384 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:42.788581 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:42.789355 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:42.789360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:42.789362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:42.789492 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:42.789875 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:42.789878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:42.789880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:42.790153 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:42.790325 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:42.790656 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 19:42:42.790811 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:42:42.794874 18189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:42:42.801280 18189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:42:42.820176 18378 heartbeater.cc:503] Master 127.2.164.126:44997 was elected leader, sending a full tablet report...
I20260926 19:42:42.834604 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.814524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.2.164.67:43551)
0926 19:42:42.827054 (+ 12530us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:42.827062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:42.827730 (+ 668us) server_negotiation.cc:407] Connection header received
0926 19:42:42.828035 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:42.828042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:42.828246 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:42.828420 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:42.829246 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.830341 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.832212 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.833512 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.833624 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.833843 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:42.834113 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 19:42:42.834275 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12373,"thread_start_us":12225,"threads_started":1}
I20260926 19:42:42.834803 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.813960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43551 (local address 127.0.0.1:53826)
0926 19:42:42.827534 (+ 13574us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:42.827550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:42.827839 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:42.828662 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:42.828667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:42.828690 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:42.829044 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.829055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.830478 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.830482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.832076 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.832088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.833691 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.833697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.833879 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.833933 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 19:42:42.834060 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 19:42:42.834270 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13384,"mutex_wait_us":1,"thread_start_us":13251,"threads_started":1}
I20260926 19:42:42.835427 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.812445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.2.164.66:36773)
0926 19:42:42.826586 (+ 14141us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:42.826594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:42.826621 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:42:42.826717 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:42.826724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:42.826960 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:42.827146 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:42.828101 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.829550 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.832376 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.833660 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.833795 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.833974 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:42.834980 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0926 19:42:42.835154 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13992,"thread_start_us":13906,"threads_started":1}
I20260926 19:42:42.835671 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.812085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:36773 (local address 127.0.0.1:36724)
0926 19:42:42.825123 (+ 13038us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:42.825195 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:42.825385 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:42.827375 (+ 1990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:42.827379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:42.827402 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:42.827913 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.827925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.829687 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.829692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.832196 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.832209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.833996 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.834002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.834201 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.834233 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 19:42:42.834310 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 19:42:42.834422 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12747,"mutex_wait_us":134,"thread_start_us":13960,"threads_started":1}
I20260926 19:42:42.835812 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.812765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.2.164.65:37793)
0926 19:42:42.826029 (+ 13264us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:42.826038 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:42.826499 (+ 461us) server_negotiation.cc:407] Connection header received
0926 19:42:42.826598 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:42.826606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:42.826863 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:42.827050 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:42.828051 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.829474 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.832953 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.834327 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.834452 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.834651 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:42.835142 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 19:42:42.835314 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13089,"thread_start_us":12931,"threads_started":1}
I20260926 19:42:42.836356 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.812371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:37793 (local address 127.0.0.1:45576)
0926 19:42:42.826251 (+ 13880us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:42.826268 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:42.826386 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:42.827322 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:42.827330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:42.827362 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:42.827859 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.827876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.829590 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.829596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.832795 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.832809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.834794 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.834801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.834962 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.834993 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:42:42.835085 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 19:42:42.835217 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13756,"mutex_wait_us":136,"thread_start_us":12340,"threads_started":1}
I20260926 19:42:42.838824 18447 tablet_service.cc:1467] Processing CreateTablet for tablet c799eeb473514d3dba9fd95cd83ac789 (DEFAULT_TABLE table=test-table [id=3b7b24b8bd8e4db4b436a9df7ca60f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:42.838915 18580 tablet_service.cc:1467] Processing CreateTablet for tablet c799eeb473514d3dba9fd95cd83ac789 (DEFAULT_TABLE table=test-table [id=3b7b24b8bd8e4db4b436a9df7ca60f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:42.839507 18313 tablet_service.cc:1467] Processing CreateTablet for tablet c799eeb473514d3dba9fd95cd83ac789 (DEFAULT_TABLE table=test-table [id=3b7b24b8bd8e4db4b436a9df7ca60f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:42:42.840004 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c799eeb473514d3dba9fd95cd83ac789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:42.840296 18580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c799eeb473514d3dba9fd95cd83ac789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:42.840728 18313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c799eeb473514d3dba9fd95cd83ac789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:42:42.851017 18663 tablet_bootstrap.cc:492] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: Bootstrap starting.
I20260926 19:42:42.853758 18664 tablet_bootstrap.cc:492] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: Bootstrap starting.
I20260926 19:42:42.854666 18663 tablet_bootstrap.cc:654] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:42.855952 18663 log.cc:824] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:42.857482 18664 tablet_bootstrap.cc:654] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:42.858675 18664 log.cc:824] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:42.858945 18663 tablet_bootstrap.cc:492] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: No bootstrap required, opened a new log
I20260926 19:42:42.859226 18663 ts_tablet_manager.cc:1397] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 19:42:42.859426 18665 tablet_bootstrap.cc:492] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: Bootstrap starting.
I20260926 19:42:42.860939 18664 tablet_bootstrap.cc:492] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: No bootstrap required, opened a new log
I20260926 19:42:42.861212 18664 ts_tablet_manager.cc:1397] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 19:42:42.862623 18665 tablet_bootstrap.cc:654] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: Neither blocks nor log segments found. Creating new log.
I20260926 19:42:42.863874 18665 log.cc:824] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: Log is configured to *not* fsync() on all Append() calls
I20260926 19:42:42.866629 18665 tablet_bootstrap.cc:492] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: No bootstrap required, opened a new log
I20260926 19:42:42.866894 18665 ts_tablet_manager.cc:1397] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260926 19:42:42.868837 18663 raft_consensus.cc:348] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } }
I20260926 19:42:42.869282 18663 raft_consensus.cc:374] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:42.869376 18663 raft_consensus.cc:729] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894784a0522c4e008d2b2114bde2cd6f, State: Initialized, Role: FOLLOWER
I20260926 19:42:42.869367 18664 raft_consensus.cc:348] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } }
I20260926 19:42:42.869733 18664 raft_consensus.cc:374] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:42.869822 18664 raft_consensus.cc:729] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f154833b09c4e179629745257d9a417, State: Initialized, Role: FOLLOWER
I20260926 19:42:42.869946 18663 consensus_queue.cc:260] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } }
I20260926 19:42:42.870350 18664 consensus_queue.cc:260] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } }
I20260926 19:42:42.871120 18512 heartbeater.cc:503] Master 127.2.164.126:44997 was elected leader, sending a full tablet report...
I20260926 19:42:42.871618 18663 ts_tablet_manager.cc:1428] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 19:42:42.873044 18664 ts_tablet_manager.cc:1428] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260926 19:42:42.876469 18665 raft_consensus.cc:348] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } }
I20260926 19:42:42.876861 18665 raft_consensus.cc:374] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:42:42.876977 18665 raft_consensus.cc:729] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 258f0ada9c8b4290b285e0a4fc438032, State: Initialized, Role: FOLLOWER
I20260926 19:42:42.877552 18665 consensus_queue.cc:260] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } }
I20260926 19:42:42.878453 18645 heartbeater.cc:503] Master 127.2.164.126:44997 was elected leader, sending a full tablet report...
I20260926 19:42:42.878960 18665 ts_tablet_manager.cc:1428] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
I20260926 19:42:42.955830 18670 raft_consensus.cc:482] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:42:42.956120 18670 raft_consensus.cc:504] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } }
I20260926 19:42:42.957719 18670 leader_election.cc:288] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 894784a0522c4e008d2b2114bde2cd6f (127.2.164.66:36773), 258f0ada9c8b4290b285e0a4fc438032 (127.2.164.67:43551)
I20260926 19:42:42.964178 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.958279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:45423 (local address 127.2.164.67:43551)
0926 19:42:42.958644 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:42.958649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:42.958782 (+ 133us) server_negotiation.cc:407] Connection header received
0926 19:42:42.958985 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:42.958989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:42.959063 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:42.959146 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:42.960023 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.960741 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.962550 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.963316 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.963459 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.963530 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:42.963811 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 19:42:42.963931 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260926 19:42:42.964326 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.958171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:48979 (local address 127.2.164.66:36773)
0926 19:42:42.958490 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:42.958497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:42.959062 (+ 565us) server_negotiation.cc:407] Connection header received
0926 19:42:42.959175 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:42.959180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:42.959251 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:42.959326 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:42:42.959895 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.960575 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.962579 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.963482 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.963615 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.963698 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:42.963964 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 19:42:42.964083 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260926 19:42:42.964344 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.957944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43551 (local address 127.2.164.65:45423)
0926 19:42:42.958690 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:42.958705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:42.958853 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:42.959449 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:42.959452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:42.959484 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:42.959883 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.959892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.960882 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.960885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.962410 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.962426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.963519 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.963525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.963672 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.963696 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 19:42:42.963782 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 19:42:42.963905 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":9,"thread_start_us":746,"threads_started":1}
I20260926 19:42:42.964576 18672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:42.957938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:36773 (local address 127.2.164.65:48979)
0926 19:42:42.958963 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0926 19:42:42.958976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 19:42:42.959077 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:42:42.959344 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:42:42.959347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:42:42.959364 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:42:42.959734 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.959742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.960702 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.960707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.962436 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:42:42.962449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:42:42.963680 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:42:42.963686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:42:42.963825 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:42.963847 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:42:42.963922 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 19:42:42.964097 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":199,"thread_start_us":370,"threads_started":1}
I20260926 19:42:42.965392 18600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c799eeb473514d3dba9fd95cd83ac789" candidate_uuid: "6f154833b09c4e179629745257d9a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "258f0ada9c8b4290b285e0a4fc438032" is_pre_election: true
I20260926 19:42:42.965492 18467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c799eeb473514d3dba9fd95cd83ac789" candidate_uuid: "6f154833b09c4e179629745257d9a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894784a0522c4e008d2b2114bde2cd6f" is_pre_election: true
I20260926 19:42:42.965855 18600 raft_consensus.cc:2454] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f154833b09c4e179629745257d9a417 in term 0.
I20260926 19:42:42.965924 18467 raft_consensus.cc:2454] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f154833b09c4e179629745257d9a417 in term 0.
I20260926 19:42:42.966493 18267 leader_election.cc:302] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f154833b09c4e179629745257d9a417, 894784a0522c4e008d2b2114bde2cd6f; no voters:
I20260926 19:42:42.966847 18670 raft_consensus.cc:2781] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:42:42.967003 18670 raft_consensus.cc:482] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:42:42.967110 18670 raft_consensus.cc:3037] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:42.969218 18670 raft_consensus.cc:504] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } }
I20260926 19:42:42.969797 18670 leader_election.cc:288] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [CANDIDATE]: Term 1 election: Requested vote from peers 894784a0522c4e008d2b2114bde2cd6f (127.2.164.66:36773), 258f0ada9c8b4290b285e0a4fc438032 (127.2.164.67:43551)
I20260926 19:42:42.970161 18467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c799eeb473514d3dba9fd95cd83ac789" candidate_uuid: "6f154833b09c4e179629745257d9a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894784a0522c4e008d2b2114bde2cd6f"
I20260926 19:42:42.970259 18600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c799eeb473514d3dba9fd95cd83ac789" candidate_uuid: "6f154833b09c4e179629745257d9a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "258f0ada9c8b4290b285e0a4fc438032"
I20260926 19:42:42.970366 18467 raft_consensus.cc:3037] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:42.970436 18600 raft_consensus.cc:3037] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:42:42.973490 18467 raft_consensus.cc:2454] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f154833b09c4e179629745257d9a417 in term 1.
I20260926 19:42:42.973490 18600 raft_consensus.cc:2454] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f154833b09c4e179629745257d9a417 in term 1.
I20260926 19:42:42.973898 18269 leader_election.cc:302] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 258f0ada9c8b4290b285e0a4fc438032, 6f154833b09c4e179629745257d9a417, 894784a0522c4e008d2b2114bde2cd6f; no voters:
I20260926 19:42:42.974220 18670 raft_consensus.cc:2781] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:42:42.974627 18670 raft_consensus.cc:686] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 1 LEADER]: Becoming Leader. State: Replica: 6f154833b09c4e179629745257d9a417, State: Running, Role: LEADER
I20260926 19:42:42.975122 18670 consensus_queue.cc:237] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } }
I20260926 19:42:42.979596 18189 catalog_manager.cc:5495] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f154833b09c4e179629745257d9a417 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "6f154833b09c4e179629745257d9a417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } health_report { overall_health: UNKNOWN } } }
W20260926 19:42:42.994791 18646 tablet.cc:2369] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:42:43.019119 18513 tablet.cc:2369] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:42:43.024667 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:43.019218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.2.164.65:37793)
0926 19:42:43.019410 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:43.019415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:43.019435 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:42:43.019794 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:43.019798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:43.019882 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:43.020025 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:42:43.020399 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:43.021114 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:43.022564 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:43.022974 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:43.023082 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:43.023573 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:42:43.024073 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:42:43.024212 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:43.024314 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 19:42:43.024394 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:42:43.032518 18467 raft_consensus.cc:1261] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f [term 1 FOLLOWER]: Refusing update from remote peer 6f154833b09c4e179629745257d9a417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:42:43.032550 18600 raft_consensus.cc:1261] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032 [term 1 FOLLOWER]: Refusing update from remote peer 6f154833b09c4e179629745257d9a417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:42:43.033571 18674 consensus_queue.cc:1035] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:42:43.033862 18670 consensus_queue.cc:1035] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 19:42:43.042562 18379 tablet.cc:2369] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:42:43.046603 18679 mvcc.cc:205] Tried to move back new op lower bound from 7333690421371285504 to 7333690421150048256. Current Snapshot: MvccSnapshot[applied={T|T < 7333690421371285504}]
I20260926 19:42:43.059444 18189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:42:43.059669 18189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:42:43.062839 18189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Sending DeleteTablet for 3 replicas of tablet c799eeb473514d3dba9fd95cd83ac789
I20260926 19:42:43.063845 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet c799eeb473514d3dba9fd95cd83ac789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:43 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:36724
I20260926 19:42:43.064183 18313 tablet_service.cc:1514] Processing DeleteTablet for tablet c799eeb473514d3dba9fd95cd83ac789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:43 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:45576
I20260926 19:42:43.064810 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 18250
I20260926 19:42:43.064857 18580 tablet_service.cc:1514] Processing DeleteTablet for tablet c799eeb473514d3dba9fd95cd83ac789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:42:43 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:53826
I20260926 19:42:43.069657 18688 tablet_replica.cc:330] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: stopping tablet replica
I20260926 19:42:43.070098 18688 raft_consensus.cc:2229] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:42:43.070811 18688 raft_consensus.cc:2258] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:42:43.072664 18688 ts_tablet_manager.cc:1905] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:42:43.077908 18689 tablet_replica.cc:330] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: stopping tablet replica
I20260926 19:42:43.078369 18689 raft_consensus.cc:2229] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:42:43.078877 18689 raft_consensus.cc:2258] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:42:43.079622 18687 tablet_replica.cc:330] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: stopping tablet replica
I20260926 19:42:43.079936 18687 raft_consensus.cc:2229] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:42:43.080035 18688 ts_tablet_manager.cc:1918] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:42:43.080214 18688 log.cc:1196] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/c799eeb473514d3dba9fd95cd83ac789
I20260926 19:42:43.080348 18687 raft_consensus.cc:2258] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:42:43.080746 18688 ts_tablet_manager.cc:1939] T c799eeb473514d3dba9fd95cd83ac789 P 6f154833b09c4e179629745257d9a417: Deleting consensus metadata
I20260926 19:42:43.082027 18687 ts_tablet_manager.cc:1905] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:42:43.082626 18173 catalog_manager.cc:4841] TS 6f154833b09c4e179629745257d9a417 (127.2.164.65:37793): tablet c799eeb473514d3dba9fd95cd83ac789 (table test-table [id=3b7b24b8bd8e4db4b436a9df7ca60f4f]) successfully deleted
I20260926 19:42:43.087010 18687 ts_tablet_manager.cc:1918] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:42:43.087157 18687 log.cc:1196] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/c799eeb473514d3dba9fd95cd83ac789
I20260926 19:42:43.087666 18687 ts_tablet_manager.cc:1939] T c799eeb473514d3dba9fd95cd83ac789 P 894784a0522c4e008d2b2114bde2cd6f: Deleting consensus metadata
I20260926 19:42:43.089305 18174 catalog_manager.cc:4841] TS 894784a0522c4e008d2b2114bde2cd6f (127.2.164.66:36773): tablet c799eeb473514d3dba9fd95cd83ac789 (table test-table [id=3b7b24b8bd8e4db4b436a9df7ca60f4f]) successfully deleted
I20260926 19:42:43.089663 18689 ts_tablet_manager.cc:1905] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:42:43.094964 18689 ts_tablet_manager.cc:1918] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:42:43.095098 18689 log.cc:1196] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/c799eeb473514d3dba9fd95cd83ac789
I20260926 19:42:43.095470 18689 ts_tablet_manager.cc:1939] T c799eeb473514d3dba9fd95cd83ac789 P 258f0ada9c8b4290b285e0a4fc438032: Deleting consensus metadata
Sep 26 19:42:43 dist-test-slave-mrw4 krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:42:43.097123 18176 catalog_manager.cc:4841] TS 258f0ada9c8b4290b285e0a4fc438032 (127.2.164.67:43551): tablet c799eeb473514d3dba9fd95cd83ac789 (table test-table [id=3b7b24b8bd8e4db4b436a9df7ca60f4f]) successfully deleted
I20260926 19:42:43.103004 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:43.065687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.2.164.65:37793)
0926 19:42:43.065857 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:43.065864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:43.085788 (+ 19924us) server_negotiation.cc:407] Connection header received
0926 19:42:43.085995 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:43.086000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:43.086095 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:43.086196 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:43.088033 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:43.089243 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:43.091259 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:43.091969 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:43.092089 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:43.097432 (+ 5343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:43.097442 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:43.097456 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:43.097486 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:43.100331 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:43.100998 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:43.101005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:43.101007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:43.101158 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:43.101549 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:43.101554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:43.101557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:43.101901 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:43.102156 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:43.102473 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 19:42:43.102689 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:42:43.103644 18373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:43.199461 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 18250
I20260926 19:42:43.214311 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 18384
Sep 26 19:42:43 dist-test-slave-mrw4 krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:42:43.229756 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:43.215083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36730 (local address 127.2.164.66:36773)
0926 19:42:43.215261 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:43.215266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:43.215286 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:42:43.215504 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:43.215510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:43.215590 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:43.215680 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:43.216999 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:43.217814 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:43.219163 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:43.219584 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:43.219693 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:43.223966 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:43.223982 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:43.223998 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:43.224034 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:43.226918 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:43.227617 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:43.227625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:43.227627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:43.227753 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:43.228105 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:43.228109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:43.228111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:43.228469 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:43.228749 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:43.229044 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 19:42:43.229472 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:42:43.230371 18507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:43.317142 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 18384
I20260926 19:42:43.330886 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 18517
Sep 26 19:42:43 dist-test-slave-mrw4 krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:42:43.346237 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:42:43.331673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.2.164.67:43551)
0926 19:42:43.331869 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 19:42:43.331876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:42:43.331897 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:42:43.332073 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:42:43.332077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:42:43.332157 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:42:43.332241 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:42:43.333671 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:43.334385 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:43.335670 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:42:43.336054 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:42:43.336162 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:42:43.340794 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:42:43.340810 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:42:43.340827 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:42:43.340864 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:42:43.343460 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:43.344163 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:43.344170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:43.344172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:43.344319 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:42:43.344684 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:42:43.344688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:42:43.344690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:42:43.345046 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:42:43.345349 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 19:42:43.345680 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 19:42:43.345896 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:42:43.346882 18640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:43.435590 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 18517
I20260926 19:42:43.451505 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 18157
I20260926 19:42:43.453178 18219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:42:43.569643 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 18157
I20260926 19:43:03.589813 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44997
--webserver_interface=127.2.164.126
--webserver_port=35213
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:44997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:03.899811 18695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:03.900115 18695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:03.900249 18695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:03.907406 18695 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 19:43:03.907521 18695 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 19:43:03.907584 18695 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 19:43:03.907655 18695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:43:03.907729 18695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:03.907820 18695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:03.907877 18695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:43:03.907950 18695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:43:03.917757 18695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:44997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=35213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:03.919050 18695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:03.920318 18695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:03.941505 18701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:03.941481 18704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:03.941447 18702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:03.941824 18695 server_base.cc:1034] running on GCE node
I20260926 19:43:03.942989 18695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:03.944545 18695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:03.945791 18695 hybrid_clock.cc:648] HybridClock initialized: now 1790451783945718 us; error 29 us; skew 500 ppm
I20260926 19:43:03.946179 18695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:03 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:03.952977 18695 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:43:03.954718 18695 webserver.cc:460] Webserver started at http://127.2.164.126:35213/ using document root <none> and password file <none>
I20260926 19:43:03.955472 18695 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:03.955571 18695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:03.961555 18695 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:43:03.964425 18711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:03.965816 18695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:43:03.965972 18695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "39ccbb9a19104808afd74ab8795ad074"
format_stamp: "Formatted at 2026-09-26 19:42:41 on dist-test-slave-mrw4"
server_key: "0428e91f327a5fb946b32f758d169012"
server_key_iv: "284a712c35936cdfe1bc3039ac6c52f1"
server_key_version: "encryptionkey@0"
I20260926 19:43:03.966998 18695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:03.980986 18695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:03.985038 18695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:03.985347 18695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:04.002910 18695 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:44997
I20260926 19:43:04.002938 18762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:44997 every 8 connection(s)
I20260926 19:43:04.004410 18695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:43:04.010706 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 18695
I20260926 19:43:04.011873 18763 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:43:04.011922 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:37793
--local_ip_for_outbound_sockets=127.2.164.65
--tserver_master_addrs=127.2.164.126:44997
--webserver_port=44255
--webserver_interface=127.2.164.65
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:43:04.016614 18763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Bootstrap starting.
I20260926 19:43:04.037650 18763 log.cc:824] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:04.050128 18763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:43:04.050617 18763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Bootstrap complete.
I20260926 19:43:04.059325 18763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:43:04.060191 18763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ccbb9a19104808afd74ab8795ad074, State: Initialized, Role: FOLLOWER
I20260926 19:43:04.060762 18763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:43:04.061002 18763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:43:04.061116 18763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:43:04.061300 18763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:43:04.064973 18763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:43:04.065492 18763 leader_election.cc:302] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39ccbb9a19104808afd74ab8795ad074; no voters:
I20260926 19:43:04.066097 18763 leader_election.cc:288] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 19:43:04.066234 18767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 19:43:04.067374 18767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 2 LEADER]: Becoming Leader. State: Replica: 39ccbb9a19104808afd74ab8795ad074, State: Running, Role: LEADER
I20260926 19:43:04.068014 18767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:43:04.068483 18763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:43:04.070482 18768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "39ccbb9a19104808afd74ab8795ad074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } } }
I20260926 19:43:04.070869 18768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:04.070928 18769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ccbb9a19104808afd74ab8795ad074. Latest consensus state: current_term: 2 leader_uuid: "39ccbb9a19104808afd74ab8795ad074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } } }
I20260926 19:43:04.071267 18769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:04.074301 18775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:43:04.080655 18775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:43:04.081995 18775 catalog_manager.cc:1259] Loaded cluster ID: 2ba1a8674f0e4f9681123c93617e1fd6
I20260926 19:43:04.082087 18775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:43:04.084718 18775 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:43:04.085813 18775 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Loaded TSK: 0
I20260926 19:43:04.107380 18775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Generated new TSK 1
I20260926 19:43:04.107685 18775 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 19:43:04.363138 18765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:04.363425 18765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:04.363561 18765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:04.370947 18765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:04.371061 18765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:04.371207 18765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:43:04.380980 18765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:37793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=44255
--tserver_master_addrs=127.2.164.126:44997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:04.382232 18765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:04.383535 18765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:04.397480 18790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:04.397526 18791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:04.398072 18793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:04.398452 18765 server_base.cc:1034] running on GCE node
I20260926 19:43:04.399181 18765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:04.400331 18765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:04.401566 18765 hybrid_clock.cc:648] HybridClock initialized: now 1790451784401509 us; error 45 us; skew 500 ppm
I20260926 19:43:04.401949 18765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:04 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451784, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:04.405833 18765 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:43:04.407532 18765 webserver.cc:460] Webserver started at http://127.2.164.65:44255/ using document root <none> and password file <none>
I20260926 19:43:04.408422 18765 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:04.408572 18765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:04.415314 18765 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:43:04.418532 18800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:04.420069 18765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:43:04.420218 18765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "6f154833b09c4e179629745257d9a417"
format_stamp: "Formatted at 2026-09-26 19:42:41 on dist-test-slave-mrw4"
server_key: "7cb42e318407f265ca07f9ba3775a532"
server_key_iv: "781edd97fee3abfb4d3c84e51d52c020"
server_key_version: "encryptionkey@0"
I20260926 19:43:04.421473 18765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:04.447037 18765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:04.451113 18765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:04.451352 18765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:04.452313 18765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:04.453673 18765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:04.453758 18765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:04.453855 18765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:04.453923 18765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:04.482692 18765 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:37793
I20260926 19:43:04.482754 18912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:37793 every 8 connection(s)
I20260926 19:43:04.484014 18765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:43:04.486963 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 18765
I20260926 19:43:04.487603 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:36773
--local_ip_for_outbound_sockets=127.2.164.66
--tserver_master_addrs=127.2.164.126:44997
--webserver_port=36599
--webserver_interface=127.2.164.66
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:43:04 dist-test-slave-mrw4 krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451784, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:04.514853 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:04.486852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:51755)
0926 19:43:04.493716 (+ 6864us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:04.493761 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:04.495488 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:04.497049 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:04.497056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:04.497595 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:04.498124 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:04.498142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:04.500330 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:04.500334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:04.501728 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:04.501741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:04.503104 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:04.503109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:04.503305 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:04.503941 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:04.503964 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:04.506129 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:04.510754 (+ 4625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:04.510771 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:04.510783 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:04.511234 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:04.511688 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:04.511691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:04.511694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:04.511893 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:04.512750 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:04.512756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:04.513129 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 19:43:04.513419 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 19:43:04.513846 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6573,"thread_start_us":6357,"threads_started":1}
I20260926 19:43:04.515241 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:04.486689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:51755 (local address 127.2.164.126:44997)
0926 19:43:04.487368 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:04.487386 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:04.493916 (+ 6530us) server_negotiation.cc:407] Connection header received
0926 19:43:04.496077 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:04.496095 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:04.496570 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:04.496962 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:04.498318 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:04.500138 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:04.501862 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:04.502847 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:04.503041 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:04.506452 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:04.506465 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:04.506488 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:04.506545 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:04.510596 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:04.511372 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:04.511381 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:04.511390 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:04.511587 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:04.511992 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:04.511997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:04.512000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:04.512595 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:04.512784 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:04.513683 (+ 899us) server_negotiation.cc:299] Negotiation successful
0926 19:43:04.514129 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":244,"threads_started":1}
I20260926 19:43:04.518038 18913 heartbeater.cc:347] Connected to a master server at 127.2.164.126:44997
I20260926 19:43:04.518350 18913 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:04.519155 18913 heartbeater.cc:511] Master 127.2.164.126:44997 requested a full tablet report, sending...
I20260926 19:43:04.521600 18728 ts_manager.cc:194] Registered new tserver with Master: 6f154833b09c4e179629745257d9a417 (127.2.164.65:37793)
I20260926 19:43:04.523064 18728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:51755
W20260926 19:43:04.801314 18917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:04.801585 18917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:04.801728 18917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:04.808699 18917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:04.808804 18917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:04.808948 18917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:43:04.818154 18917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:36773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=36599
--tserver_master_addrs=127.2.164.126:44997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:04.819301 18917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:04.820506 18917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:04.832545 18922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:04.832579 18923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:04.832610 18925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:04.833359 18917 server_base.cc:1034] running on GCE node
I20260926 19:43:04.833896 18917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:04.834738 18917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:04.835939 18917 hybrid_clock.cc:648] HybridClock initialized: now 1790451784835877 us; error 55 us; skew 500 ppm
I20260926 19:43:04.836230 18917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:04 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451784, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:04.839027 18917 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:43:04.840363 18917 webserver.cc:460] Webserver started at http://127.2.164.66:36599/ using document root <none> and password file <none>
I20260926 19:43:04.840967 18917 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:04.841089 18917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:04.846014 18917 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:43:04.848440 18932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:04.849697 18917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:43:04.849846 18917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "894784a0522c4e008d2b2114bde2cd6f"
format_stamp: "Formatted at 2026-09-26 19:42:42 on dist-test-slave-mrw4"
server_key: "427be354d4e2440cf154059259739baf"
server_key_iv: "3d2998f0e96d78a43b04137b12af08f2"
server_key_version: "encryptionkey@0"
I20260926 19:43:04.850739 18917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:04.864575 18917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:04.867993 18917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:04.868221 18917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:04.869123 18917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:04.870365 18917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:04.870463 18917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:04.870558 18917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:04.870622 18917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:04.899211 18917 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:36773
I20260926 19:43:04.899295 19044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:36773 every 8 connection(s)
I20260926 19:43:04.900594 18917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:43:04.909538 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 18917
I20260926 19:43:04.910276 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:43551
--local_ip_for_outbound_sockets=127.2.164.67
--tserver_master_addrs=127.2.164.126:44997
--webserver_port=43145
--webserver_interface=127.2.164.67
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:43:04 dist-test-slave-mrw4 krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451784, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:04.920220 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:04.902957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:36379 (local address 127.2.164.126:44997)
0926 19:43:04.903171 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:04.903177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:04.904294 (+ 1117us) server_negotiation.cc:407] Connection header received
0926 19:43:04.906082 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:04.906086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:04.906157 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:04.906285 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:04.907675 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:04.908371 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:04.910232 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:04.910709 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:04.910809 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:04.914402 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:04.914410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:04.914415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:04.914455 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:04.917235 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:04.917972 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:04.917977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:04.917979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:04.918101 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:04.918567 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:04.918571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:04.918573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:04.918846 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:04.918959 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:04.919664 (+ 705us) server_negotiation.cc:299] Negotiation successful
0926 19:43:04.919889 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 19:43:04.920950 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:04.903465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:36379)
0926 19:43:04.904138 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:04.904175 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:04.905807 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:04.906457 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:04.906465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:04.906955 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:04.907510 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:04.907529 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:04.908489 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:04.908493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:04.910097 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:04.910112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:04.910932 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:04.910940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:04.911200 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:04.911853 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:04.911878 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:04.914172 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:04.917388 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:04.917394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:04.917407 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:04.917830 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:04.918196 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:04.918201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:04.918205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:04.918437 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:04.918959 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:04.918966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:04.919289 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 19:43:04.919534 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 19:43:04.919901 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":245,"threads_started":1}
I20260926 19:43:04.922330 19045 heartbeater.cc:347] Connected to a master server at 127.2.164.126:44997
I20260926 19:43:04.922649 19045 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:04.923466 19045 heartbeater.cc:511] Master 127.2.164.126:44997 requested a full tablet report, sending...
I20260926 19:43:04.924944 18728 ts_manager.cc:194] Registered new tserver with Master: 894784a0522c4e008d2b2114bde2cd6f (127.2.164.66:36773)
I20260926 19:43:04.925778 18728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:36379
W20260926 19:43:05.224681 19048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:05.224953 19048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:05.225086 19048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:05.232319 19048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:05.232419 19048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:05.232559 19048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:43:05.241933 19048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:43551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=43145
--tserver_master_addrs=127.2.164.126:44997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:05.243132 19048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:05.244349 19048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:05.256415 19056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:05.256458 19054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:05.256486 19053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:05.257123 19048 server_base.cc:1034] running on GCE node
I20260926 19:43:05.257684 19048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:05.258525 19048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:05.259722 19048 hybrid_clock.cc:648] HybridClock initialized: now 1790451785259675 us; error 37 us; skew 500 ppm
I20260926 19:43:05.260010 19048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:05 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451785, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:05.262933 19048 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:43:05.264207 19048 webserver.cc:460] Webserver started at http://127.2.164.67:43145/ using document root <none> and password file <none>
I20260926 19:43:05.264864 19048 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:05.264973 19048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:05.270092 19048 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 19:43:05.272658 19063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:05.273964 19048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:43:05.274107 19048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "258f0ada9c8b4290b285e0a4fc438032"
format_stamp: "Formatted at 2026-09-26 19:42:42 on dist-test-slave-mrw4"
server_key: "9ece45e3717b8219a0904072990aeee3"
server_key_iv: "773be7891651db28076ee0c9af041d6c"
server_key_version: "encryptionkey@0"
I20260926 19:43:05.275002 19048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:05.300158 19048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:05.303905 19048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:05.304136 19048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:05.305038 19048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:05.306337 19048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:05.306427 19048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:05.306543 19048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:05.306610 19048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:05.335127 19048 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:43551
I20260926 19:43:05.335180 19175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:43551 every 8 connection(s)
I20260926 19:43:05.336542 19048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:43:05.346295 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 19048
Sep 26 19:43:05 dist-test-slave-mrw4 krb5kdc[18144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451785, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:05.356166 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.338905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:41877 (local address 127.2.164.126:44997)
0926 19:43:05.339097 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:05.339103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:05.340193 (+ 1090us) server_negotiation.cc:407] Connection header received
0926 19:43:05.341967 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:05.341971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:05.342037 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:05.342130 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:05.343627 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.344333 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.345949 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.346517 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.346640 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.350185 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:05.350193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:05.350196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:05.350225 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:05.352893 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:05.353712 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:05.353716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:05.353720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:05.353876 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:05.354361 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:05.354366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:05.354370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:05.354812 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:05.354968 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:05.355594 (+ 626us) server_negotiation.cc:299] Negotiation successful
0926 19:43:05.355803 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:43:05.357059 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.339383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:41877)
0926 19:43:05.340039 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:05.340075 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:05.341718 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:05.342385 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:05.342393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:05.342920 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:05.343472 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.343490 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.344478 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.344482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.345799 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.345812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.346710 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.346715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.346960 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.347770 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:05.347801 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:05.349992 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:05.353073 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:05.353081 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:05.353098 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:05.353567 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:05.353994 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:05.353998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:05.354002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:05.354235 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:05.354945 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:05.354951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:05.355252 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 19:43:05.355488 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 19:43:05.355840 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":245,"threads_started":1}
I20260926 19:43:05.358587 19176 heartbeater.cc:347] Connected to a master server at 127.2.164.126:44997
I20260926 19:43:05.358877 19176 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:05.359645 19176 heartbeater.cc:511] Master 127.2.164.126:44997 requested a full tablet report, sending...
I20260926 19:43:05.361210 18728 ts_manager.cc:194] Registered new tserver with Master: 258f0ada9c8b4290b285e0a4fc438032 (127.2.164.67:43551)
I20260926 19:43:05.361866 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.347190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.2.164.126:44997)
0926 19:43:05.348244 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:05.348248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:05.348260 (+ 12us) server_negotiation.cc:407] Connection header received
0926 19:43:05.348292 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:05.348296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:05.348343 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:05.348415 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:05.350140 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.351022 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.352453 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.352954 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.353069 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.356210 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:05.356217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:05.356222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:05.356257 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:05.359183 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:05.359989 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:05.359992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:05.359994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:05.360100 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:05.360503 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:05.360508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:05.360511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:05.360802 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:05.361072 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:05.361300 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 19:43:05.361508 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":272,"threads_started":1}
I20260926 19:43:05.362258 18728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:41877
I20260926 19:43:05.364259 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:43:05.370440 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.365085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.2.164.126:44997)
0926 19:43:05.365267 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:05.365272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:05.365694 (+ 422us) server_negotiation.cc:407] Connection header received
0926 19:43:05.365886 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:05.365890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:05.366005 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:05.366115 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:43:05.366524 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.367170 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.368593 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.369009 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.369109 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.369654 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:43:05.369835 (+ 181us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 19:43:05.370176 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52048: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 19:43:05.370535 19179 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52048: token expired
W20260926 19:43:05.371025 18649 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.164.126:44997)
I20260926 19:43:05.383256 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.371443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.2.164.126:44997)
0926 19:43:05.371585 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:05.371590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:05.371643 (+ 53us) server_negotiation.cc:407] Connection header received
0926 19:43:05.371786 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:05.371791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:05.371845 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:05.371916 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:05.373188 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.373903 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.375182 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.375571 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.375675 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.378243 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:05.378249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:05.378252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:05.378280 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:05.380718 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:05.381424 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:05.381428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:05.381430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:05.381573 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:05.382033 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:05.382040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:05.382042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:05.382352 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:05.382553 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:05.382803 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 19:43:05.382966 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 19:43:05.386327 18728 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:43:05.393371 18728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:43:05.424149 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.404406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.2.164.66:36773)
0926 19:43:05.415635 (+ 11229us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:05.415642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:05.415669 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:43:05.415744 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:05.415751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:05.415956 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:05.416148 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:05.417019 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.418485 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.421618 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.422716 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.422848 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.423044 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:05.423675 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 19:43:05.423849 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11079,"thread_start_us":10970,"threads_started":1}
I20260926 19:43:05.424211 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.404069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:36773 (local address 127.0.0.1:44068)
0926 19:43:05.414426 (+ 10357us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:05.414461 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:05.414623 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:05.416509 (+ 1886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:05.416516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:05.416546 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:05.416848 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.416856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.418650 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.418656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.421266 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.421278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.423201 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.423208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.423422 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.423484 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 19:43:05.423615 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 19:43:05.423796 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10121,"mutex_wait_us":2,"thread_start_us":10709,"threads_started":1}
I20260926 19:43:05.424852 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.404473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43551 (local address 127.0.0.1:50686)
0926 19:43:05.415048 (+ 10575us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:05.415065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:05.415174 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:05.418438 (+ 3264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:05.418444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:05.418481 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:05.419022 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.419035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.420624 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.420628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.422662 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.422676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.424317 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.424324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.424464 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.424488 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 19:43:05.424536 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 19:43:05.424616 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10443,"mutex_wait_us":21,"thread_start_us":14292,"threads_started":1}
I20260926 19:43:05.425866 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.405082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.2.164.67:43551)
0926 19:43:05.417870 (+ 12788us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:05.417878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:05.417908 (+ 30us) server_negotiation.cc:407] Connection header received
0926 19:43:05.417999 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:05.418006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:05.418227 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:05.418398 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:05.419208 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.420498 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.422805 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.424084 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.425067 (+ 983us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.425288 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:05.425357 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 19:43:05.425557 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12628,"thread_start_us":12483,"threads_started":1}
I20260926 19:43:05.428161 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.404677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:37793 (local address 127.0.0.1:37506)
0926 19:43:05.419072 (+ 14395us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:05.419090 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:05.419280 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:05.420082 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:05.420086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:05.420109 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:05.420467 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.420479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.422309 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.422314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.425513 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.425525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.427326 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.427332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.427635 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.427669 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:43:05.427745 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 19:43:05.427865 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14261,"thread_start_us":9377,"threads_started":1}
I20260926 19:43:05.428419 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.405597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.2.164.65:37793)
0926 19:43:05.417405 (+ 11808us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:05.417413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:05.419181 (+ 1768us) server_negotiation.cc:407] Connection header received
0926 19:43:05.419470 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:05.419478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:05.419678 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:05.419827 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:05.420663 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.422173 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.425639 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.426698 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.426980 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.427167 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:05.427809 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 19:43:05.427968 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11643,"thread_start_us":11546,"threads_started":1}
I20260926 19:43:05.428318 18980 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7bab7c4ea345d2af185ea0dfc76e46 (DEFAULT_TABLE table=test-table [id=1d6beecc5e4544679e4ff5a55a9549fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:05.429757 18980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7bab7c4ea345d2af185ea0dfc76e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:05.429318 19111 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7bab7c4ea345d2af185ea0dfc76e46 (DEFAULT_TABLE table=test-table [id=1d6beecc5e4544679e4ff5a55a9549fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:05.430655 19111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7bab7c4ea345d2af185ea0dfc76e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:05.431032 18848 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7bab7c4ea345d2af185ea0dfc76e46 (DEFAULT_TABLE table=test-table [id=1d6beecc5e4544679e4ff5a55a9549fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:05.432158 18848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7bab7c4ea345d2af185ea0dfc76e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:05.442988 19192 tablet_bootstrap.cc:492] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: Bootstrap starting.
I20260926 19:43:05.446738 19192 tablet_bootstrap.cc:654] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:05.447741 19192 log.cc:824] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:05.448231 19194 tablet_bootstrap.cc:492] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: Bootstrap starting.
I20260926 19:43:05.449807 19193 tablet_bootstrap.cc:492] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: Bootstrap starting.
I20260926 19:43:05.451822 19192 tablet_bootstrap.cc:492] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: No bootstrap required, opened a new log
I20260926 19:43:05.452099 19192 ts_tablet_manager.cc:1397] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260926 19:43:05.453296 19193 tablet_bootstrap.cc:654] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:05.454527 19193 log.cc:824] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:05.458006 19194 tablet_bootstrap.cc:654] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:05.459170 19194 log.cc:824] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:05.459277 19193 tablet_bootstrap.cc:492] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: No bootstrap required, opened a new log
I20260926 19:43:05.459497 19193 ts_tablet_manager.cc:1397] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260926 19:43:05.461719 19192 raft_consensus.cc:348] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } }
I20260926 19:43:05.462080 19194 tablet_bootstrap.cc:492] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: No bootstrap required, opened a new log
I20260926 19:43:05.462174 19192 raft_consensus.cc:374] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:05.462270 19192 raft_consensus.cc:729] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f154833b09c4e179629745257d9a417, State: Initialized, Role: FOLLOWER
I20260926 19:43:05.462311 19194 ts_tablet_manager.cc:1397] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20260926 19:43:05.462899 19192 consensus_queue.cc:260] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } }
I20260926 19:43:05.466917 19193 raft_consensus.cc:348] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } }
I20260926 19:43:05.467214 19193 raft_consensus.cc:374] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:05.467322 19193 raft_consensus.cc:729] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894784a0522c4e008d2b2114bde2cd6f, State: Initialized, Role: FOLLOWER
I20260926 19:43:05.467972 18913 heartbeater.cc:503] Master 127.2.164.126:44997 was elected leader, sending a full tablet report...
I20260926 19:43:05.467932 19193 consensus_queue.cc:260] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } }
I20260926 19:43:05.468580 19192 ts_tablet_manager.cc:1428] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260926 19:43:05.470237 19045 heartbeater.cc:503] Master 127.2.164.126:44997 was elected leader, sending a full tablet report...
I20260926 19:43:05.470811 19193 ts_tablet_manager.cc:1428] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260926 19:43:05.472324 19194 raft_consensus.cc:348] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } }
I20260926 19:43:05.472762 19194 raft_consensus.cc:374] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:05.472853 19194 raft_consensus.cc:729] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 258f0ada9c8b4290b285e0a4fc438032, State: Initialized, Role: FOLLOWER
I20260926 19:43:05.473395 19194 consensus_queue.cc:260] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } }
I20260926 19:43:05.474982 19176 heartbeater.cc:503] Master 127.2.164.126:44997 was elected leader, sending a full tablet report...
I20260926 19:43:05.475417 19194 ts_tablet_manager.cc:1428] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260926 19:43:05.485888 18914 tablet.cc:2369] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:43:05.588176 19177 tablet.cc:2369] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:43:05.652495 19046 tablet.cc:2369] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:43:05.804972 19198 raft_consensus.cc:482] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:43:05.805289 19198 raft_consensus.cc:504] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } }
I20260926 19:43:05.806963 19198 leader_election.cc:288] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 894784a0522c4e008d2b2114bde2cd6f (127.2.164.66:36773), 258f0ada9c8b4290b285e0a4fc438032 (127.2.164.67:43551)
I20260926 19:43:05.815963 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.807488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:34523 (local address 127.2.164.66:36773)
0926 19:43:05.807687 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:05.807692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:05.808416 (+ 724us) server_negotiation.cc:407] Connection header received
0926 19:43:05.808632 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:05.808638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:05.808717 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:05.808820 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:05.809697 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.810761 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.813648 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.814830 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.814988 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.815061 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:05.815573 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 19:43:05.815702 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:43:05.815963 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.807497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:60581 (local address 127.2.164.67:43551)
0926 19:43:05.807683 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:05.807690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:05.808394 (+ 704us) server_negotiation.cc:407] Connection header received
0926 19:43:05.808632 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:05.808638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:05.808717 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:05.808820 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:05.809694 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.810842 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.813648 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.814830 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.814998 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.815061 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:05.815569 (+ 508us) server_negotiation.cc:299] Negotiation successful
0926 19:43:05.815702 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:43:05.816154 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.807360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:43551 (local address 127.2.164.65:60581)
0926 19:43:05.808291 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:05.808305 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:05.808477 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:05.808881 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:05.808901 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:05.808938 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:05.809493 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.809513 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.810955 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.810959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.813472 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.813489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.815093 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.815104 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.815373 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.815443 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 19:43:05.815532 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 19:43:05.815694 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":183,"spinlock_wait_cycles":65024,"thread_start_us":545,"threads_started":1}
I20260926 19:43:05.816529 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.807294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:36773 (local address 127.2.164.65:34523)
0926 19:43:05.808239 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:05.808266 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:05.808437 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:05.808884 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:05.808889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:05.808932 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:05.809554 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.809563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.810928 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.810933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.813498 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:05.813507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.815116 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.815121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:05.815378 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.815439 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 19:43:05.815530 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 19:43:05.815694 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":224,"thread_start_us":588,"threads_started":1}
I20260926 19:43:05.817140 19131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7bab7c4ea345d2af185ea0dfc76e46" candidate_uuid: "6f154833b09c4e179629745257d9a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "258f0ada9c8b4290b285e0a4fc438032" is_pre_election: true
I20260926 19:43:05.817514 19000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7bab7c4ea345d2af185ea0dfc76e46" candidate_uuid: "6f154833b09c4e179629745257d9a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894784a0522c4e008d2b2114bde2cd6f" is_pre_election: true
I20260926 19:43:05.817780 19131 raft_consensus.cc:2454] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f154833b09c4e179629745257d9a417 in term 0.
I20260926 19:43:05.818014 19000 raft_consensus.cc:2454] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f154833b09c4e179629745257d9a417 in term 0.
I20260926 19:43:05.818434 18804 leader_election.cc:302] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 258f0ada9c8b4290b285e0a4fc438032, 6f154833b09c4e179629745257d9a417; no voters:
I20260926 19:43:05.818789 19198 raft_consensus.cc:2781] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:43:05.818944 19198 raft_consensus.cc:482] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:43:05.819043 19198 raft_consensus.cc:3037] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:05.821090 19198 raft_consensus.cc:504] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } }
I20260926 19:43:05.821650 19198 leader_election.cc:288] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [CANDIDATE]: Term 1 election: Requested vote from peers 894784a0522c4e008d2b2114bde2cd6f (127.2.164.66:36773), 258f0ada9c8b4290b285e0a4fc438032 (127.2.164.67:43551)
I20260926 19:43:05.822082 19000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7bab7c4ea345d2af185ea0dfc76e46" candidate_uuid: "6f154833b09c4e179629745257d9a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894784a0522c4e008d2b2114bde2cd6f"
I20260926 19:43:05.822101 19131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7bab7c4ea345d2af185ea0dfc76e46" candidate_uuid: "6f154833b09c4e179629745257d9a417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "258f0ada9c8b4290b285e0a4fc438032"
I20260926 19:43:05.822261 19000 raft_consensus.cc:3037] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:05.822293 19131 raft_consensus.cc:3037] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:05.824354 19131 raft_consensus.cc:2454] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f154833b09c4e179629745257d9a417 in term 1.
I20260926 19:43:05.824430 19000 raft_consensus.cc:2454] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f154833b09c4e179629745257d9a417 in term 1.
I20260926 19:43:05.824720 18804 leader_election.cc:302] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 258f0ada9c8b4290b285e0a4fc438032, 6f154833b09c4e179629745257d9a417; no voters:
I20260926 19:43:05.825022 19198 raft_consensus.cc:2781] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:43:05.825466 19198 raft_consensus.cc:686] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 1 LEADER]: Becoming Leader. State: Replica: 6f154833b09c4e179629745257d9a417, State: Running, Role: LEADER
I20260926 19:43:05.825938 19198 consensus_queue.cc:237] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } }
I20260926 19:43:05.830183 18728 catalog_manager.cc:5495] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f154833b09c4e179629745257d9a417 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "6f154833b09c4e179629745257d9a417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f154833b09c4e179629745257d9a417" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 37793 } health_report { overall_health: HEALTHY } } }
I20260926 19:43:05.878592 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.873368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.2.164.65:37793)
0926 19:43:05.873577 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:05.873582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:05.873602 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:43:05.873773 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:05.873777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:05.873864 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:05.873987 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:43:05.874349 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.875060 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.876448 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.876895 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.876997 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.877538 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:43:05.878040 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:43:05.878167 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:05.878238 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 19:43:05.878336 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:43:05.886885 19131 raft_consensus.cc:1261] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032 [term 1 FOLLOWER]: Refusing update from remote peer 6f154833b09c4e179629745257d9a417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:05.886946 19000 raft_consensus.cc:1261] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f [term 1 FOLLOWER]: Refusing update from remote peer 6f154833b09c4e179629745257d9a417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:05.887804 19203 consensus_queue.cc:1035] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894784a0522c4e008d2b2114bde2cd6f" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:43:05.888155 19198 consensus_queue.cc:1035] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "258f0ada9c8b4290b285e0a4fc438032" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:43:05.897477 19205 mvcc.cc:205] Tried to move back new op lower bound from 7333690514983399424 to 7333690514746576896. Current Snapshot: MvccSnapshot[applied={T|T < 7333690514983399424}]
I20260926 19:43:05.917354 18727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:05.917608 18727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:05.920984 18727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Sending DeleteTablet for 3 replicas of tablet 7f7bab7c4ea345d2af185ea0dfc76e46
I20260926 19:43:05.922047 18848 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f7bab7c4ea345d2af185ea0dfc76e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:05 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:37506
I20260926 19:43:05.922040 19111 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f7bab7c4ea345d2af185ea0dfc76e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:05 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:50686
I20260926 19:43:05.922094 18980 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f7bab7c4ea345d2af185ea0dfc76e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:05 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:44068
I20260926 19:43:05.924553 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 18765
I20260926 19:43:05.936894 19216 tablet_replica.cc:330] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: stopping tablet replica
I20260926 19:43:05.938024 19216 raft_consensus.cc:2229] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:05.939066 19215 tablet_replica.cc:330] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: stopping tablet replica
I20260926 19:43:05.939133 19216 raft_consensus.cc:2258] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:05.940941 19214 tablet_replica.cc:330] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: stopping tablet replica
I20260926 19:43:05.941504 19214 raft_consensus.cc:2229] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:43:05.941586 19216 ts_tablet_manager.cc:1905] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:05.941861 19215 raft_consensus.cc:2229] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:05.942174 19214 raft_consensus.cc:2258] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:05.942662 19215 raft_consensus.cc:2258] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:05.944357 19214 ts_tablet_manager.cc:1905] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:05.946188 19215 ts_tablet_manager.cc:1905] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:05.948405 19216 ts_tablet_manager.cc:1918] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:05.948592 19216 log.cc:1196] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/7f7bab7c4ea345d2af185ea0dfc76e46
I20260926 19:43:05.949151 19216 ts_tablet_manager.cc:1939] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 894784a0522c4e008d2b2114bde2cd6f: Deleting consensus metadata
I20260926 19:43:05.950981 18713 catalog_manager.cc:4841] TS 894784a0522c4e008d2b2114bde2cd6f (127.2.164.66:36773): tablet 7f7bab7c4ea345d2af185ea0dfc76e46 (table test-table [id=1d6beecc5e4544679e4ff5a55a9549fd]) successfully deleted
I20260926 19:43:05.951483 19214 ts_tablet_manager.cc:1918] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:05.951650 19214 log.cc:1196] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/7f7bab7c4ea345d2af185ea0dfc76e46
I20260926 19:43:05.952195 19214 ts_tablet_manager.cc:1939] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 6f154833b09c4e179629745257d9a417: Deleting consensus metadata
I20260926 19:43:05.953367 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:05.925379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.2.164.65:37793)
0926 19:43:05.925564 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:05.925571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:05.931969 (+ 6398us) server_negotiation.cc:407] Connection header received
0926 19:43:05.932144 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:05.932151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:05.932237 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:05.932359 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:05.934575 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.935847 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.938614 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:05.939178 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:05.939293 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:05.943757 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:05.943770 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:05.943787 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:05.943829 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:05.947677 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:05.949422 (+ 1745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:05.949434 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:05.949438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:05.949615 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:05.951586 (+ 1971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:05.951591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:05.951595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:05.952105 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:05.952457 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:05.952755 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 19:43:05.952973 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:43:05.953827 18712 catalog_manager.cc:4841] TS 6f154833b09c4e179629745257d9a417 (127.2.164.65:37793): tablet 7f7bab7c4ea345d2af185ea0dfc76e46 (table test-table [id=1d6beecc5e4544679e4ff5a55a9549fd]) successfully deleted
I20260926 19:43:05.954314 18908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:05.956892 19215 ts_tablet_manager.cc:1918] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:05.957057 19215 log.cc:1196] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/7f7bab7c4ea345d2af185ea0dfc76e46
I20260926 19:43:05.957604 19215 ts_tablet_manager.cc:1939] T 7f7bab7c4ea345d2af185ea0dfc76e46 P 258f0ada9c8b4290b285e0a4fc438032: Deleting consensus metadata
I20260926 19:43:05.958971 18715 catalog_manager.cc:4841] TS 258f0ada9c8b4290b285e0a4fc438032 (127.2.164.67:43551): tablet 7f7bab7c4ea345d2af185ea0dfc76e46 (table test-table [id=1d6beecc5e4544679e4ff5a55a9549fd]) successfully deleted
I20260926 19:43:06.050536 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 18765
I20260926 19:43:06.065980 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 18917
I20260926 19:43:06.079164 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:06.066712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.2.164.66:36773)
0926 19:43:06.066902 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:06.066907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:06.066924 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:43:06.067071 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:06.067074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:06.067136 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:06.067222 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:06.068506 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:06.069309 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:06.070681 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:06.071097 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:06.071188 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:06.073791 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:06.073812 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:06.073831 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:06.073865 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:06.076569 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:06.077256 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:06.077265 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:06.077267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:06.077439 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:06.077775 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:06.077779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:06.077781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:06.078220 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:06.078458 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:06.078668 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 19:43:06.078824 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:43:06.079873 19040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:06.166309 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 18917
I20260926 19:43:06.180122 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 19048
I20260926 19:43:06.193459 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:06.180804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.2.164.67:43551)
0926 19:43:06.180989 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:06.180993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:06.181011 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:43:06.181192 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:06.181198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:06.181279 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:06.181371 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:06.182692 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:06.183407 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:06.184630 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:06.185028 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:06.185130 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:06.187859 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:06.187874 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:06.187893 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:06.187926 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:06.190701 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:06.191448 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:06.191456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:06.191458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:06.191613 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:06.192001 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:06.192005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:06.192007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:06.192374 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:06.192597 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:06.192860 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 19:43:06.193018 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:43:06.194044 19171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:06.279534 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 19048
I20260926 19:43:06.292763 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 18695
I20260926 19:43:06.293907 18758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:06.409808 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 18695
I20260926 19:43:26.436928 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44997
--webserver_interface=127.2.164.126
--webserver_port=35213
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:44997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:26.760978 19222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:26.761335 19222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:26.761469 19222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:26.768462 19222 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 19:43:26.768558 19222 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 19:43:26.768623 19222 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 19:43:26.768687 19222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:43:26.768759 19222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:26.768857 19222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:26.768919 19222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:43:26.768991 19222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:43:26.778650 19222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:44997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=35213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:26.779884 19222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:26.781183 19222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:26.792280 19228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:26.792395 19227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:26.792594 19230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:26.792887 19222 server_base.cc:1034] running on GCE node
I20260926 19:43:26.793669 19222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:26.794996 19222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:26.796235 19222 hybrid_clock.cc:648] HybridClock initialized: now 1790451806796158 us; error 41 us; skew 500 ppm
I20260926 19:43:26.796548 19222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:26 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451806, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:26.799540 19222 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:43:26.800936 19222 webserver.cc:460] Webserver started at http://127.2.164.126:35213/ using document root <none> and password file <none>
I20260926 19:43:26.801602 19222 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:26.801702 19222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:26.806866 19222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 19:43:26.809577 19237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:26.810827 19222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:43:26.811003 19222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "39ccbb9a19104808afd74ab8795ad074"
format_stamp: "Formatted at 2026-09-26 19:42:41 on dist-test-slave-mrw4"
server_key: "0428e91f327a5fb946b32f758d169012"
server_key_iv: "284a712c35936cdfe1bc3039ac6c52f1"
server_key_version: "encryptionkey@0"
I20260926 19:43:26.811908 19222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:26.833564 19222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:26.837412 19222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:26.837659 19222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:26.854208 19222 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:44997
I20260926 19:43:26.854238 19288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:44997 every 8 connection(s)
I20260926 19:43:26.855595 19222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:43:26.861451 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 19222
I20260926 19:43:26.862517 19289 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:43:26.862519 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:37793
--local_ip_for_outbound_sockets=127.2.164.65
--tserver_master_addrs=127.2.164.126:44997
--webserver_port=44255
--webserver_interface=127.2.164.65
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:43:26.866801 19289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Bootstrap starting.
I20260926 19:43:26.887274 19289 log.cc:824] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:26.902659 19289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:43:26.903163 19289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Bootstrap complete.
I20260926 19:43:26.911881 19289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:43:26.912727 19289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ccbb9a19104808afd74ab8795ad074, State: Initialized, Role: FOLLOWER
I20260926 19:43:26.913393 19289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:43:26.913605 19289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:43:26.913714 19289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:43:26.913867 19289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 2 FOLLOWER]: Advancing to term 3
I20260926 19:43:26.917846 19289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:43:26.918306 19289 leader_election.cc:302] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39ccbb9a19104808afd74ab8795ad074; no voters:
I20260926 19:43:26.918907 19289 leader_election.cc:288] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 19:43:26.919039 19293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 19:43:26.920132 19293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [term 3 LEADER]: Becoming Leader. State: Replica: 39ccbb9a19104808afd74ab8795ad074, State: Running, Role: LEADER
I20260926 19:43:26.920763 19293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } }
I20260926 19:43:26.921011 19289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:43:26.923125 19295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ccbb9a19104808afd74ab8795ad074. Latest consensus state: current_term: 3 leader_uuid: "39ccbb9a19104808afd74ab8795ad074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } } }
I20260926 19:43:26.923524 19295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:26.923883 19294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "39ccbb9a19104808afd74ab8795ad074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ccbb9a19104808afd74ab8795ad074" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44997 } } }
I20260926 19:43:26.924181 19294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074 [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:26.934315 19302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:43:26.942448 19302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:43:26.943949 19302 catalog_manager.cc:1259] Loaded cluster ID: 2ba1a8674f0e4f9681123c93617e1fd6
I20260926 19:43:26.944051 19302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:43:26.948146 19302 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:43:26.949836 19302 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Loaded TSK: 0
I20260926 19:43:26.949980 19302 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 39ccbb9a19104808afd74ab8795ad074: Loaded TSK: 1
F20260926 19:43:26.951121 19302 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff509b249ad google::LogMessage::Fail() at ??:0
@ 0x7ff509b2788a google::LogMessage::SendToLog() at ??:0
@ 0x7ff509b244de google::LogMessage::Flush() at ??:0
@ 0x7ff509b25a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff50d8887d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff50d87773e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff51f5531c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff51f552783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff51f5de2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7ff51f5de0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7ff51f6285e9 std::function<>::operator()() at ??:0
@ 0x7ff51f556cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff51f554c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff51f5dc610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff51f5dc439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff50c770345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff50c77391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff50c773799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff50c74eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff50fa286db start_thread at ??:0
@ 0x7ff5089be61f clone at ??:0
W20260926 19:43:27.216915 19291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:27.217262 19291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:27.217401 19291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:27.224674 19291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:27.224799 19291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:27.224942 19291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:43:27.234896 19291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:37793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=44255
--tserver_master_addrs=127.2.164.126:44997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:27.236264 19291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:27.237574 19291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:27.250087 19339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:27.250141 19336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:27.250141 19337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:27.251005 19291 server_base.cc:1034] running on GCE node
I20260926 19:43:27.251513 19291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:27.252377 19291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:27.253571 19291 hybrid_clock.cc:648] HybridClock initialized: now 1790451807253516 us; error 49 us; skew 500 ppm
I20260926 19:43:27.253875 19291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:27 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451807, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:27.256892 19291 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:43:27.258337 19291 webserver.cc:460] Webserver started at http://127.2.164.65:44255/ using document root <none> and password file <none>
I20260926 19:43:27.258965 19291 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:27.259059 19291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:27.264243 19291 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 19:43:27.266924 19346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:27.268185 19291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:43:27.268326 19291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "6f154833b09c4e179629745257d9a417"
format_stamp: "Formatted at 2026-09-26 19:42:41 on dist-test-slave-mrw4"
server_key: "7cb42e318407f265ca07f9ba3775a532"
server_key_iv: "781edd97fee3abfb4d3c84e51d52c020"
server_key_version: "encryptionkey@0"
I20260926 19:43:27.269320 19291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:27.288697 19291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:27.292689 19291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:27.292940 19291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:27.293879 19291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:27.295156 19291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:27.295246 19291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:27.295354 19291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:27.295421 19291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:27.325101 19291 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:37793
I20260926 19:43:27.325198 19458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:37793 every 8 connection(s)
I20260926 19:43:27.326480 19291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:43:27.327499 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 19291
I20260926 19:43:27.328265 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:36773
--local_ip_for_outbound_sockets=127.2.164.66
--tserver_master_addrs=127.2.164.126:44997
--webserver_port=36599
--webserver_interface=127.2.164.66
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:43:27.331779 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:27.329645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:48911)
0926 19:43:27.330422 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:27.330692 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":280,"threads_started":1}
W20260926 19:43:27.332490 19459 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:44997 (0 consecutive failures): Network error: Failed to ping master at 127.2.164.126:44997: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
I20260926 19:43:27.333318 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:27.332949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:41209)
0926 19:43:27.333095 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:27.333203 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 19:43:27.334182 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:27.333855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:39005)
0926 19:43:27.333983 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:27.334064 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 19:43:27.334415 19459 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 19:43:27.646623 19462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:27.646888 19462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:27.647011 19462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:27.654124 19462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:27.654224 19462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:27.654373 19462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:43:27.663829 19462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:36773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=36599
--tserver_master_addrs=127.2.164.126:44997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:27.665040 19462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:27.666337 19462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:27.679395 19467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:27.679308 19470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:27.679306 19468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:27.679852 19462 server_base.cc:1034] running on GCE node
I20260926 19:43:27.680545 19462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:27.681444 19462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:27.682636 19462 hybrid_clock.cc:648] HybridClock initialized: now 1790451807682591 us; error 30 us; skew 500 ppm
I20260926 19:43:27.682909 19462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:27 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451807, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:27.685819 19462 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:43:27.687178 19462 webserver.cc:460] Webserver started at http://127.2.164.66:36599/ using document root <none> and password file <none>
I20260926 19:43:27.687830 19462 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:27.687932 19462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:27.693040 19462 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:43:27.695777 19477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:27.697029 19462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:43:27.697185 19462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "894784a0522c4e008d2b2114bde2cd6f"
format_stamp: "Formatted at 2026-09-26 19:42:42 on dist-test-slave-mrw4"
server_key: "427be354d4e2440cf154059259739baf"
server_key_iv: "3d2998f0e96d78a43b04137b12af08f2"
server_key_version: "encryptionkey@0"
I20260926 19:43:27.698120 19462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:27.711617 19462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:27.715632 19462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:27.715885 19462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:27.716797 19462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:27.718137 19462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:27.718219 19462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:27.718314 19462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:27.718379 19462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:27.747139 19462 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:36773
I20260926 19:43:27.747212 19589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:36773 every 8 connection(s)
I20260926 19:43:27.748526 19462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:43:27.750523 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 19462
I20260926 19:43:27.751295 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:43551
--local_ip_for_outbound_sockets=127.2.164.67
--tserver_master_addrs=127.2.164.126:44997
--webserver_port=43145
--webserver_interface=127.2.164.67
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:43:27.753743 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:27.751840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:40209)
0926 19:43:27.752604 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:27.752821 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":306,"threads_started":1}
W20260926 19:43:27.754310 19590 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:44997 (0 consecutive failures): Network error: Failed to ping master at 127.2.164.126:44997: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
I20260926 19:43:27.755059 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:27.754779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:51877)
0926 19:43:27.754902 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:27.754963 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 19:43:27.755805 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:27.755510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:38207)
0926 19:43:27.755625 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:27.755693 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 19:43:27.756006 19590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 19:43:28.059067 19593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:28.059345 19593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:28.059484 19593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:28.066656 19593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:28.066795 19593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:28.066936 19593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:43:28.076999 19593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:43551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=43145
--tserver_master_addrs=127.2.164.126:44997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:28.078343 19593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:28.079643 19593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:28.092118 19601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:28.092089 19598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:28.092146 19599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:28.092594 19593 server_base.cc:1034] running on GCE node
I20260926 19:43:28.093304 19593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:28.094180 19593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:28.095381 19593 hybrid_clock.cc:648] HybridClock initialized: now 1790451808095323 us; error 56 us; skew 500 ppm
I20260926 19:43:28.095664 19593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:28 dist-test-slave-mrw4 krb5kdc[18144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451808, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:28.098609 19593 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:43:28.099978 19593 webserver.cc:460] Webserver started at http://127.2.164.67:43145/ using document root <none> and password file <none>
I20260926 19:43:28.100595 19593 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:28.100706 19593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:28.105778 19593 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:43:28.108444 19608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:28.109803 19593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:43:28.109949 19593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "258f0ada9c8b4290b285e0a4fc438032"
format_stamp: "Formatted at 2026-09-26 19:42:42 on dist-test-slave-mrw4"
server_key: "9ece45e3717b8219a0904072990aeee3"
server_key_iv: "773be7891651db28076ee0c9af041d6c"
server_key_version: "encryptionkey@0"
I20260926 19:43:28.110852 19593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:28.121491 19593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:28.125263 19593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:28.125509 19593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:28.126402 19593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:28.127601 19593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:28.127703 19593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:28.127810 19593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:28.127883 19593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:28.158004 19593 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:43551
I20260926 19:43:28.158102 19720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:43551 every 8 connection(s)
I20260926 19:43:28.159389 19593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:43:28.162518 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 19593
I20260926 19:43:28.164227 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:28.162299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:36597)
0926 19:43:28.163122 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:28.163373 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":310,"threads_started":1}
W20260926 19:43:28.164932 19721 heartbeater.cc:644] Failed to heartbeat to 127.2.164.126:44997 (0 consecutive failures): Network error: Failed to ping master at 127.2.164.126:44997: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
I20260926 19:43:28.165716 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:28.165408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:43397)
0926 19:43:28.165552 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:28.165616 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 19:43:28.166481 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:28.166173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:39609)
0926 19:43:28.166304 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:28.166381 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 19:43:28.166682 19721 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 19:43:28.335772 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:28.334980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:50935)
0926 19:43:28.335428 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:28.335534 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":214,"threads_started":1}
I20260926 19:43:28.757365 19726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:28.756521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:33663)
0926 19:43:28.756994 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:28.757091 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":214,"threads_started":1}
I20260926 19:43:29.167917 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:29.167152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:54429)
0926 19:43:29.167596 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:29.167683 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":211,"threads_started":1}
I20260926 19:43:29.337359 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:29.336605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:33393)
0926 19:43:29.337076 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:29.337157 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":241,"threads_started":1}
I20260926 19:43:29.759068 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:29.758298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:54071)
0926 19:43:29.758788 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:29.758888 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":226,"threads_started":1}
I20260926 19:43:30.169611 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:30.168785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:59927)
0926 19:43:30.169347 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:30.169436 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":285,"threads_started":1}
I20260926 19:43:30.338891 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:30.338177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:41291)
0926 19:43:30.338636 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:30.338718 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":246,"threads_started":1}
I20260926 19:43:30.760650 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:30.759919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:59161)
0926 19:43:30.760400 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:30.760484 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":232,"threads_started":1}
I20260926 19:43:31.171200 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:31.170550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:58353)
0926 19:43:31.170960 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:31.171039 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":217,"threads_started":1}
I20260926 19:43:31.340452 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:31.339710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:37349)
0926 19:43:31.340159 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:31.340242 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":233,"threads_started":1}
I20260926 19:43:31.762125 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:31.761474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:45625)
0926 19:43:31.761863 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:31.761945 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":190,"threads_started":1}
I20260926 19:43:32.172787 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:32.172008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:36579)
0926 19:43:32.172511 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:32.172598 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":230,"threads_started":1}
I20260926 19:43:32.342073 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:32.341358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:59077)
0926 19:43:32.341817 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:32.341899 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":230,"threads_started":1}
I20260926 19:43:32.763700 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:32.762963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:59455)
0926 19:43:32.763450 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:32.763536 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":229,"threads_started":1}
I20260926 19:43:33.174445 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:33.173693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:51833)
0926 19:43:33.174179 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:33.174261 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":265,"threads_started":1}
I20260926 19:43:33.343631 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:33.342946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:36549)
0926 19:43:33.343379 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:33.343468 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":213,"threads_started":1}
I20260926 19:43:33.765319 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:33.764574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:54029)
0926 19:43:33.765012 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:33.765101 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":220,"threads_started":1}
I20260926 19:43:34.175977 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:34.175266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:52123)
0926 19:43:34.175714 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:34.175805 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":235,"threads_started":1}
I20260926 19:43:34.345357 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:34.344552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:39017)
0926 19:43:34.345021 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:34.345098 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":210,"threads_started":1}
I20260926 19:43:34.766974 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:34.766226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:46179)
0926 19:43:34.766681 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:34.766762 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":203,"threads_started":1}
I20260926 19:43:35.177655 19745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:35.176815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:54493)
0926 19:43:35.177333 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:35.177436 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":235,"threads_started":1}
I20260926 19:43:35.346901 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:35.346226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:33551)
0926 19:43:35.346653 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:35.346733 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260926 19:43:35.768522 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:35.767802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:36147)
0926 19:43:35.768278 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:35.768362 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":244,"threads_started":1}
I20260926 19:43:36.179139 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:36.178467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:47803)
0926 19:43:36.178891 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:36.178970 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":200,"threads_started":1}
I20260926 19:43:36.348526 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:36.347767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:40833)
0926 19:43:36.348259 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:36.348339 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":250,"threads_started":1}
I20260926 19:43:36.770318 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:36.769514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:46675)
0926 19:43:36.770009 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:36.770110 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":231,"threads_started":1}
I20260926 19:43:37.180858 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:37.180126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:36063)
0926 19:43:37.180599 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:37.180689 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":222,"threads_started":1}
I20260926 19:43:37.350291 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:37.349545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:51229)
0926 19:43:37.350028 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:37.350118 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":251,"threads_started":1}
I20260926 19:43:37.772106 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:37.771343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:43247)
0926 19:43:37.771828 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:37.771922 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":265,"threads_started":1}
I20260926 19:43:38.182425 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:38.181725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:34951)
0926 19:43:38.182154 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:38.182236 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":233,"threads_started":1}
I20260926 19:43:38.351867 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:38.351190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:56993)
0926 19:43:38.351595 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:38.351687 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":196,"threads_started":1}
I20260926 19:43:38.773741 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:38.772954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:56701)
0926 19:43:38.773436 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:38.773545 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":241,"threads_started":1}
I20260926 19:43:39.184098 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:39.183342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:34935)
0926 19:43:39.183831 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:39.183920 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":255,"threads_started":1}
I20260926 19:43:39.353595 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:39.352773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:46899)
0926 19:43:39.353324 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:39.353411 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":292,"threads_started":1}
I20260926 19:43:39.775295 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:39.774617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:50969)
0926 19:43:39.775055 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:39.775139 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":211,"threads_started":1}
I20260926 19:43:40.185698 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:40.184993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:59227)
0926 19:43:40.185447 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:40.185548 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":247,"threads_started":1}
I20260926 19:43:40.355335 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:40.354524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:46509)
0926 19:43:40.355050 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:40.355153 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":262,"threads_started":1}
I20260926 19:43:40.776846 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:40.776135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:52723)
0926 19:43:40.776573 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:40.776669 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":209,"threads_started":1}
I20260926 19:43:41.187361 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:41.186543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:59675)
0926 19:43:41.187036 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:41.187152 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":243,"threads_started":1}
I20260926 19:43:41.356832 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:41.356138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:41777)
0926 19:43:41.356574 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:41.356654 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":223,"threads_started":1}
I20260926 19:43:41.778469 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:41.777777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:58141)
0926 19:43:41.778218 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:41.778296 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":233,"threads_started":1}
I20260926 19:43:42.188956 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:42.188202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:50509)
0926 19:43:42.188692 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:42.188779 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":235,"threads_started":1}
I20260926 19:43:42.358516 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:42.357800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.65:49249)
0926 19:43:42.358240 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:42.358325 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":229,"threads_started":1}
I20260926 19:43:42.780200 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:42.779473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.66:49005)
0926 19:43:42.779930 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:42.780017 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":223,"threads_started":1}
I20260926 19:43:43.167207 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 19291
I20260926 19:43:43.187332 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:43.168138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.2.164.65:37793)
0926 19:43:43.168671 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:43.168689 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:43.168884 (+ 195us) server_negotiation.cc:407] Connection header received
0926 19:43:43.170724 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:43.170734 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:43.171152 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:43.171434 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:43.172753 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:43.174068 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:43.175285 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:43.175935 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:43.176072 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:43.181939 (+ 5867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:43.181947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:43.181962 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:43.182001 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:43.184580 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:43.185233 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:43.185240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:43.185246 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:43.185406 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:43.185746 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:43.185748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:43.185750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:43.186227 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:43.186442 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:43.186733 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 19:43:43.187039 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":209,"threads_started":1}
I20260926 19:43:43.188616 19454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:43.190836 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:43.190081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:44997 (local address 127.2.164.67:36415)
0926 19:43:43.190528 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:43.190640 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.126:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":230,"threads_started":1}
I20260926 19:43:43.269887 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 19291
I20260926 19:43:43.283411 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 19462
I20260926 19:43:43.300483 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:43.284350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.2.164.66:36773)
0926 19:43:43.284852 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:43.284867 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:43.284904 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:43:43.286431 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:43.286441 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:43.286871 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:43.287205 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:43.288559 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:43.289832 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:43.291031 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:43.291713 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:43.291879 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:43.294364 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:43.294375 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:43.294395 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:43.294440 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:43.297420 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:43.298052 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:43.298058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:43.298064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:43.298228 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:43.298588 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:43.298592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:43.298594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:43.299090 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:43.299365 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:43.299820 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 19:43:43.300209 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":222,"threads_started":1}
I20260926 19:43:43.302042 19585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:43.382562 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 19462
I20260926 19:43:43.396958 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 19593
I20260926 19:43:43.413695 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:43.397887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.2.164.67:43551)
0926 19:43:43.398455 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:43.398470 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:43.398512 (+ 42us) server_negotiation.cc:407] Connection header received
0926 19:43:43.400078 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:43.400089 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:43.400561 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:43.400888 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:43.402286 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:43.403455 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:43.404649 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:43.405361 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:43.405527 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:43.407915 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:43.407922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:43.407937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:43.407971 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:43.410629 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:43.411248 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:43.411254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:43.411260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:43.411421 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:43.411779 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:43.411783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:43.411785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:43.412273 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:43.412542 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:43.412939 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 19:43:43.413375 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":256,"threads_started":1}
I20260926 19:43:43.415094 19716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:43.500360 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 19593
2026-09-26T19:43:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63912 ms)
[----------] 30 tests from SecurityITest (282779 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:43:43 dist-test-slave-mrw4 krb5kdc[19778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:43:43 dist-test-slave-mrw4 krb5kdc[19778](info): set up 2 sockets
Sep 26 19:43:43 dist-test-slave-mrw4 krb5kdc[19778](info): commencing operation
krb5kdc: starting...
W20260926 19:43:44.513235 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:43:44 dist-test-slave-mrw4 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:43:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:43:44.728940 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36087
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:36087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:45.031924 19791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:45.032208 19791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:45.032338 19791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:45.039265 19791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:43:45.039372 19791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:45.039450 19791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:45.039521 19791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:43:45.039582 19791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:43:45.049063 19791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:36087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:45.050346 19791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:45.051591 19791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:45.062582 19796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:45.062615 19799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:45.062687 19797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:45.063218 19791 server_base.cc:1034] running on GCE node
I20260926 19:43:45.063903 19791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:45.065140 19791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:45.066380 19791 hybrid_clock.cc:648] HybridClock initialized: now 1790451825066332 us; error 35 us; skew 500 ppm
I20260926 19:43:45.066696 19791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:45 dist-test-slave-mrw4 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:45.069897 19791 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:43:45.071216 19791 webserver.cc:460] Webserver started at http://127.2.164.126:36239/ using document root <none> and password file <none>
I20260926 19:43:45.071825 19791 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:45.071923 19791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:45.072206 19791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:45.074999 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "6166c5d6aff249c5a40a4a46e9eeeeea"
format_stamp: "Formatted at 2026-09-26 19:43:45 on dist-test-slave-mrw4"
server_key: "2f9dac633b4bdcc049c1c85e9a31cebd"
server_key_iv: "e3419f7d505cdd99319c876c96fcad7c"
server_key_version: "encryptionkey@0"
I20260926 19:43:45.075695 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "6166c5d6aff249c5a40a4a46e9eeeeea"
format_stamp: "Formatted at 2026-09-26 19:43:45 on dist-test-slave-mrw4"
server_key: "2f9dac633b4bdcc049c1c85e9a31cebd"
server_key_iv: "e3419f7d505cdd99319c876c96fcad7c"
server_key_version: "encryptionkey@0"
I20260926 19:43:45.080497 19791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:43:45.083796 19806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:45.084951 19791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:43:45.085095 19791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "6166c5d6aff249c5a40a4a46e9eeeeea"
format_stamp: "Formatted at 2026-09-26 19:43:45 on dist-test-slave-mrw4"
server_key: "2f9dac633b4bdcc049c1c85e9a31cebd"
server_key_iv: "e3419f7d505cdd99319c876c96fcad7c"
server_key_version: "encryptionkey@0"
I20260926 19:43:45.085307 19791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:45.096942 19791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:45.100721 19791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:45.100930 19791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:45.117388 19791 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:36087
I20260926 19:43:45.117414 19857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:36087 every 8 connection(s)
I20260926 19:43:45.118647 19791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:43:45.122097 19858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:45.127892 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 19791
I20260926 19:43:45.128087 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:43:45.128681 2705 external_mini_cluster.cc:1428] Setting key 05b786491161f6ea63ebe274b01be497
I20260926 19:43:45.131757 19858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea: Bootstrap starting.
I20260926 19:43:45.134963 19858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:45.136080 19858 log.cc:824] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:45.138876 19858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea: No bootstrap required, opened a new log
Sep 26 19:43:45 dist-test-slave-mrw4 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:45.147049 19858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6166c5d6aff249c5a40a4a46e9eeeeea" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36087 } }
I20260926 19:43:45.147394 19858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:45.147491 19858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6166c5d6aff249c5a40a4a46e9eeeeea, State: Initialized, Role: FOLLOWER
I20260926 19:43:45.148180 19858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6166c5d6aff249c5a40a4a46e9eeeeea" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36087 } }
I20260926 19:43:45.148478 19858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:43:45.148651 19858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:43:45.148819 19858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:45.150898 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:45.130498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.2.164.126:36087)
0926 19:43:45.131184 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:45.131195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:45.131233 (+ 38us) server_negotiation.cc:407] Connection header received
0926 19:43:45.132843 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:45.132866 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:45.133262 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:45.133559 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:45.135223 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:45.136549 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:45.138242 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:45.138882 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:45.139047 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:45.143406 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:45.143416 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:45.143429 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:45.143467 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:45.146439 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:45.147139 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:45.147145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:45.147151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:45.147301 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:45.147663 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:45.147668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:45.147671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:45.148411 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:45.148711 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:45.149144 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 19:43:45.149599 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":300,"threads_started":1}
I20260926 19:43:45.151989 19858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6166c5d6aff249c5a40a4a46e9eeeeea" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36087 } }
I20260926 19:43:45.152496 19858 leader_election.cc:302] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6166c5d6aff249c5a40a4a46e9eeeeea; no voters:
I20260926 19:43:45.153046 19858 leader_election.cc:288] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:43:45.153239 19863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:43:45.154508 19863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [term 1 LEADER]: Becoming Leader. State: Replica: 6166c5d6aff249c5a40a4a46e9eeeeea, State: Running, Role: LEADER
I20260926 19:43:45.154830 19858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:43:45.154989 19863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6166c5d6aff249c5a40a4a46e9eeeeea" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36087 } }
I20260926 19:43:45.157577 19865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6166c5d6aff249c5a40a4a46e9eeeeea. Latest consensus state: current_term: 1 leader_uuid: "6166c5d6aff249c5a40a4a46e9eeeeea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6166c5d6aff249c5a40a4a46e9eeeeea" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36087 } } }
I20260926 19:43:45.157934 19865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:45.158433 19864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6166c5d6aff249c5a40a4a46e9eeeeea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6166c5d6aff249c5a40a4a46e9eeeeea" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36087 } } }
I20260926 19:43:45.158694 19864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:45.159557 19872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:43:45.163965 19872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:43:45.171936 19872 catalog_manager.cc:1347] Generated new cluster ID: 35671ea6f7f044198b2e39be62e326ff
I20260926 19:43:45.172040 19872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:43:45.184432 19872 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:43:45.186216 19872 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:43:45.197342 19872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea: Generated new TSK 0
I20260926 19:43:45.198148 19872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:43:45.254758 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36087
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:45.563045 19884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:45.563297 19884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:45.563393 19884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:45.570441 19884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:45.570531 19884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:45.570643 19884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:43:45.579751 19884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:45.580914 19884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:45.582175 19884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:45.594609 19889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:45.594636 19892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:45.594766 19890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:45.595907 19884 server_base.cc:1034] running on GCE node
I20260926 19:43:45.596495 19884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:45.597420 19884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:45.598631 19884 hybrid_clock.cc:648] HybridClock initialized: now 1790451825598564 us; error 54 us; skew 500 ppm
I20260926 19:43:45.598922 19884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:45 dist-test-slave-mrw4 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:45.601902 19884 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:43:45.603204 19884 webserver.cc:460] Webserver started at http://127.2.164.65:42577/ using document root <none> and password file <none>
I20260926 19:43:45.603829 19884 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:45.603950 19884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:45.604236 19884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:45.607038 19884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "3ced521f5cf840289794815022d2d3e5"
format_stamp: "Formatted at 2026-09-26 19:43:45 on dist-test-slave-mrw4"
server_key: "8747bcb469c0b9ae61ab5628ef13a02b"
server_key_iv: "f219ebfc9c6938f389e70c38d01ea262"
server_key_version: "encryptionkey@0"
I20260926 19:43:45.607756 19884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "3ced521f5cf840289794815022d2d3e5"
format_stamp: "Formatted at 2026-09-26 19:43:45 on dist-test-slave-mrw4"
server_key: "8747bcb469c0b9ae61ab5628ef13a02b"
server_key_iv: "f219ebfc9c6938f389e70c38d01ea262"
server_key_version: "encryptionkey@0"
I20260926 19:43:45.612560 19884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 19:43:45.615805 19899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:45.617017 19884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:43:45.617239 19884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "3ced521f5cf840289794815022d2d3e5"
format_stamp: "Formatted at 2026-09-26 19:43:45 on dist-test-slave-mrw4"
server_key: "8747bcb469c0b9ae61ab5628ef13a02b"
server_key_iv: "f219ebfc9c6938f389e70c38d01ea262"
server_key_version: "encryptionkey@0"
I20260926 19:43:45.617408 19884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:45.639040 19884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:45.643101 19884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:45.643342 19884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:45.644218 19884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:45.645457 19884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:45.645532 19884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:45.645643 19884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:45.645694 19884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:45.675679 19884 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:43797
I20260926 19:43:45.675740 20011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:43797 every 8 connection(s)
I20260926 19:43:45.676863 19884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:43:45.687088 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 19884
I20260926 19:43:45.687252 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:43:45.687985 2705 external_mini_cluster.cc:1428] Setting key ad6d969e43ea93844b817c02c5398a01
Sep 26 19:43:45 dist-test-slave-mrw4 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:45.695673 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:45.679301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:49151 (local address 127.2.164.126:36087)
0926 19:43:45.679693 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:45.679697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:45.680609 (+ 912us) server_negotiation.cc:407] Connection header received
0926 19:43:45.682427 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:45.682434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:45.682512 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:45.682614 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:45.684061 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:45.684789 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:45.686296 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:45.686679 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:45.686786 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:45.690389 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:45.690394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:45.690397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:45.690427 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:45.692832 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:45.693613 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:45.693617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:45.693619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:45.693736 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:45.694147 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:45.694151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:45.694153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:45.694424 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:45.694532 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:45.695264 (+ 732us) server_negotiation.cc:299] Negotiation successful
0926 19:43:45.695422 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":197,"threads_started":1}
I20260926 19:43:45.696810 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:45.679784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36087 (local address 127.2.164.65:49151)
0926 19:43:45.680442 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:45.680478 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:45.682167 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:45.682856 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:45.682865 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:45.683361 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:45.683890 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:45.683907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:45.684935 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:45.684939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:45.686161 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:45.686173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:45.686883 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:45.686891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:45.687102 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:45.687764 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:45.687785 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:45.690153 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:45.692960 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:45.692968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:45.692982 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:45.693503 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:45.693821 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:45.693826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:45.693829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:45.694036 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:45.694533 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:45.694540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:45.694910 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 19:43:45.695179 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 19:43:45.695630 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":252,"threads_started":1}
I20260926 19:43:45.698283 20012 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36087
I20260926 19:43:45.698541 20012 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:45.699169 20012 heartbeater.cc:511] Master 127.2.164.126:36087 requested a full tablet report, sending...
I20260926 19:43:45.701076 19823 ts_manager.cc:194] Registered new tserver with Master: 3ced521f5cf840289794815022d2d3e5 (127.2.164.65:43797)
I20260926 19:43:45.702469 19823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:49151
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:43:45.739763 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36087
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:46.045689 20018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:46.045966 20018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:46.046109 20018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:46.053115 20018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:46.053251 20018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:46.053404 20018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:43:46.062878 20018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:46.064080 20018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:46.065343 20018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:46.077942 20023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:46.077971 20024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:46.078879 20018 server_base.cc:1034] running on GCE node
W20260926 19:43:46.079046 20026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:46.079672 20018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:46.080549 20018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:46.081736 20018 hybrid_clock.cc:648] HybridClock initialized: now 1790451826081678 us; error 54 us; skew 500 ppm
I20260926 19:43:46.082015 20018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:46 dist-test-slave-mrw4 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:46.084885 20018 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:43:46.086303 20018 webserver.cc:460] Webserver started at http://127.2.164.66:39923/ using document root <none> and password file <none>
I20260926 19:43:46.086917 20018 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:46.087018 20018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:46.087289 20018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:46.090129 20018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d"
format_stamp: "Formatted at 2026-09-26 19:43:46 on dist-test-slave-mrw4"
server_key: "6e6c9a9dba43afe6d4c5a7a64d258184"
server_key_iv: "da958231db6d9f99f9099f1d4ddbd5b2"
server_key_version: "encryptionkey@0"
I20260926 19:43:46.090830 20018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d"
format_stamp: "Formatted at 2026-09-26 19:43:46 on dist-test-slave-mrw4"
server_key: "6e6c9a9dba43afe6d4c5a7a64d258184"
server_key_iv: "da958231db6d9f99f9099f1d4ddbd5b2"
server_key_version: "encryptionkey@0"
I20260926 19:43:46.095417 20018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:43:46.098498 20033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:46.099685 20018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 19:43:46.099817 20018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d"
format_stamp: "Formatted at 2026-09-26 19:43:46 on dist-test-slave-mrw4"
server_key: "6e6c9a9dba43afe6d4c5a7a64d258184"
server_key_iv: "da958231db6d9f99f9099f1d4ddbd5b2"
server_key_version: "encryptionkey@0"
I20260926 19:43:46.099972 20018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:46.118364 20018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:46.122289 20018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:46.122534 20018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:46.123435 20018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:46.124617 20018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:46.124697 20018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:46.124810 20018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:46.124858 20018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:46.153614 20018 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:41413
I20260926 19:43:46.153669 20145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:41413 every 8 connection(s)
I20260926 19:43:46.154834 20018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:43:46.163455 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 20018
I20260926 19:43:46.163636 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:43:46.164186 2705 external_mini_cluster.cc:1428] Setting key 4446b0b7906985ccfeef8d8c670fabae
Sep 26 19:43:46 dist-test-slave-mrw4 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:46.175132 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.157262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:51601 (local address 127.2.164.126:36087)
0926 19:43:46.157452 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:46.157459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:46.158611 (+ 1152us) server_negotiation.cc:407] Connection header received
0926 19:43:46.160372 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:46.160405 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:46.160483 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:46.160569 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:46.162098 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.162834 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.164993 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.165462 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.165556 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.169505 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:46.169510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:46.169513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:46.169545 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:46.172029 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:46.172837 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:46.172842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:46.172844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:46.172979 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:46.173448 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:46.173452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:46.173455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:46.173739 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:46.173853 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:46.174578 (+ 725us) server_negotiation.cc:299] Negotiation successful
0926 19:43:46.174820 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:43:46.175817 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.157783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36087 (local address 127.2.164.66:51601)
0926 19:43:46.158457 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:46.158497 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:46.160111 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:46.160825 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:46.160834 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:46.161398 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:46.161931 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.161948 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.162989 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.162993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.164863 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.164876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.165686 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.165693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.165939 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.166553 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:46.166588 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:46.169294 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:46.172169 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:46.172177 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:46.172191 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:46.172686 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:46.173080 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:46.173084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:46.173087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:46.173331 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:46.173850 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:46.173858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:46.174232 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 19:43:46.174445 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 19:43:46.174798 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":244,"threads_started":1}
I20260926 19:43:46.177089 20146 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36087
I20260926 19:43:46.177378 20146 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:46.178035 20146 heartbeater.cc:511] Master 127.2.164.126:36087 requested a full tablet report, sending...
I20260926 19:43:46.179364 19823 ts_manager.cc:194] Registered new tserver with Master: ac6f44fafb784c6bb0b069c10d2b2c8d (127.2.164.66:41413)
I20260926 19:43:46.180137 19823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:51601
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:43:46.212764 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36087
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:46.514967 20151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:46.515232 20151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:46.515352 20151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:46.522423 20151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:46.522516 20151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:46.522655 20151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:43:46.531807 20151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:46.532991 20151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:46.534313 20151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:46.546801 20156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:46.546778 20157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:46.546859 20159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:46.547305 20151 server_base.cc:1034] running on GCE node
I20260926 19:43:46.547879 20151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:46.548729 20151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:46.549906 20151 hybrid_clock.cc:648] HybridClock initialized: now 1790451826549852 us; error 49 us; skew 500 ppm
I20260926 19:43:46.550184 20151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:46 dist-test-slave-mrw4 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:46.552973 20151 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:43:46.554320 20151 webserver.cc:460] Webserver started at http://127.2.164.67:33757/ using document root <none> and password file <none>
I20260926 19:43:46.554942 20151 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:46.555035 20151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:46.555300 20151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:46.558080 20151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "e3182c88e6df49d09167f902afb72f5e"
format_stamp: "Formatted at 2026-09-26 19:43:46 on dist-test-slave-mrw4"
server_key: "066042d368aa5e88ea256e0549d5d489"
server_key_iv: "e384c66f2af98540de015976d5407088"
server_key_version: "encryptionkey@0"
I20260926 19:43:46.558771 20151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "e3182c88e6df49d09167f902afb72f5e"
format_stamp: "Formatted at 2026-09-26 19:43:46 on dist-test-slave-mrw4"
server_key: "066042d368aa5e88ea256e0549d5d489"
server_key_iv: "e384c66f2af98540de015976d5407088"
server_key_version: "encryptionkey@0"
I20260926 19:43:46.563488 20151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:43:46.566675 20166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:46.567881 20151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 19:43:46.568001 20151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "e3182c88e6df49d09167f902afb72f5e"
format_stamp: "Formatted at 2026-09-26 19:43:46 on dist-test-slave-mrw4"
server_key: "066042d368aa5e88ea256e0549d5d489"
server_key_iv: "e384c66f2af98540de015976d5407088"
server_key_version: "encryptionkey@0"
I20260926 19:43:46.568173 20151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:46.587292 20151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:46.591228 20151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:46.591473 20151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:46.592363 20151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:46.593642 20151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:46.593712 20151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:46.593822 20151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:46.593868 20151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:46.622924 20151 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:44313
I20260926 19:43:46.622994 20278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:44313 every 8 connection(s)
I20260926 19:43:46.624092 20151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:43:46.632208 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 20151
I20260926 19:43:46.632359 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:43:46.632768 2705 external_mini_cluster.cc:1428] Setting key 2c4a68f9428074a2c00f442f63fffea3
Sep 26 19:43:46 dist-test-slave-mrw4 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:46.642514 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.626519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:49387 (local address 127.2.164.126:36087)
0926 19:43:46.626714 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:46.626728 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:46.627858 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 19:43:46.629659 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:46.629663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:46.629738 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:46.629814 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:46.631223 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.631992 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.633702 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.634201 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.634313 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.637490 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:46.637495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:46.637498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:46.637530 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:46.639750 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:46.640394 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:46.640399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:46.640401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:46.640510 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:46.640899 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:46.640904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:46.640908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:46.641225 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:46.641354 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:46.641977 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 19:43:46.642197 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:43:46.643366 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.627009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36087 (local address 127.2.164.67:49387)
0926 19:43:46.627705 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:46.627741 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:46.629412 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:46.630019 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:46.630027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:46.630540 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:46.631097 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.631114 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.632135 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.632139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.633570 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.633582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.634412 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.634419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.634634 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.635246 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:46.635272 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:46.637295 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:46.639867 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:46.639876 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:46.639891 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:46.640276 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:46.640606 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:46.640611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:46.640615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:46.640789 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:46.641339 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:46.641347 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:46.641638 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 19:43:46.641859 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 19:43:46.642334 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":270,"threads_started":1}
I20260926 19:43:46.644598 20279 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36087
I20260926 19:43:46.644867 20279 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:46.645525 20279 heartbeater.cc:511] Master 127.2.164.126:36087 requested a full tablet report, sending...
I20260926 19:43:46.646838 19822 ts_manager.cc:194] Registered new tserver with Master: e3182c88e6df49d09167f902afb72f5e (127.2.164.67:44313)
I20260926 19:43:46.647704 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:43:46.647790 19822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:49387
I20260926 19:43:46.664920 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.652608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.2.164.126:36087)
0926 19:43:46.652792 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:46.652799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:46.653093 (+ 294us) server_negotiation.cc:407] Connection header received
0926 19:43:46.653246 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:46.653249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:46.653303 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:46.653375 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:46.654826 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.655747 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.657577 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.657981 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.658083 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.660267 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:46.660272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:46.660275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:46.660308 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:46.662504 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:46.663078 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:46.663083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:46.663087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:46.663210 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:46.663563 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:46.663568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:46.663572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:46.663868 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:46.664085 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:46.664431 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 19:43:46.664626 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:43:46.668725 19822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:43:46.679741 19822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:43:46.705474 20012 heartbeater.cc:503] Master 127.2.164.126:36087 was elected leader, sending a full tablet report...
I20260926 19:43:46.707166 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.694022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.2.164.66:41413)
0926 19:43:46.694529 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:46.694537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:46.694564 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:43:46.694658 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:46.694665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:46.694915 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:46.695092 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:46.695978 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.697680 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.704212 (+ 6532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.705642 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.705768 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.705970 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:46.706687 (+ 717us) server_negotiation.cc:299] Negotiation successful
0926 19:43:46.706876 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":527,"threads_started":1}
I20260926 19:43:46.708225 20291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.693451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:41413 (local address 127.0.0.1:52530)
0926 19:43:46.694089 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:46.694132 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:46.694295 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:46.695361 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:46.695368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:46.695398 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:46.695774 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.695788 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.697847 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.697853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.704056 (+ 6203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.704070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.706113 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.706119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.706406 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.706470 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 19:43:46.707594 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0926 19:43:46.707813 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":41,"thread_start_us":351,"threads_started":1}
I20260926 19:43:46.710490 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.694450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:44313 (local address 127.0.0.1:60980)
0926 19:43:46.702838 (+ 8388us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:46.702858 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:46.702968 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:46.703468 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:46.703472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:46.703494 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:46.703872 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.703884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.705781 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.705786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.707813 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.707827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.709799 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.709804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.709911 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.709932 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 19:43:46.710008 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 19:43:46.710123 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8275,"mutex_wait_us":17,"thread_start_us":8144,"threads_started":1}
I20260926 19:43:46.710529 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.694963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60980 (local address 127.2.164.67:44313)
0926 19:43:46.695690 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:46.695699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:46.702964 (+ 7265us) server_negotiation.cc:407] Connection header received
0926 19:43:46.703093 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:46.703100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:46.703271 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:46.703453 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:46.704059 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.705617 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.707996 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.709354 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.709468 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.709692 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:46.710073 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 19:43:46.710235 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":386,"threads_started":1}
I20260926 19:43:46.712648 20081 tablet_service.cc:1467] Processing CreateTablet for tablet 41e1c6461d2e4e03b7d25e4580e86cde (DEFAULT_TABLE table=test-table [id=76d96a3620cb43b69fd6d4be132494f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:46.713910 20214 tablet_service.cc:1467] Processing CreateTablet for tablet 41e1c6461d2e4e03b7d25e4580e86cde (DEFAULT_TABLE table=test-table [id=76d96a3620cb43b69fd6d4be132494f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:46.714540 20081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e1c6461d2e4e03b7d25e4580e86cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:46.715018 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.693558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43797 (local address 127.0.0.1:36898)
0926 19:43:46.694785 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:46.694801 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:46.694905 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:46.702197 (+ 7292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:46.702201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:46.702226 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:46.702693 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.702706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.704525 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.704531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.708467 (+ 3936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.708484 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.714288 (+ 5804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.714294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.714462 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.714494 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 19:43:46.714569 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 19:43:46.714700 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"thread_start_us":1158,"threads_started":1}
I20260926 19:43:46.715323 20214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e1c6461d2e4e03b7d25e4580e86cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:46.715981 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.694405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.2.164.65:43797)
0926 19:43:46.701619 (+ 7214us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:46.701628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:46.701658 (+ 30us) server_negotiation.cc:407] Connection header received
0926 19:43:46.701757 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:46.701766 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:46.702011 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:46.702207 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:46.702926 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.704421 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.708658 (+ 4237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.714064 (+ 5406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.715231 (+ 1167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.715414 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:46.715490 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 19:43:46.715639 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7055,"thread_start_us":7782,"threads_started":1}
I20260926 19:43:46.719426 19947 tablet_service.cc:1467] Processing CreateTablet for tablet 41e1c6461d2e4e03b7d25e4580e86cde (DEFAULT_TABLE table=test-table [id=76d96a3620cb43b69fd6d4be132494f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:46.720736 19947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41e1c6461d2e4e03b7d25e4580e86cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:46.727972 20297 tablet_bootstrap.cc:492] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: Bootstrap starting.
I20260926 19:43:46.729869 20298 tablet_bootstrap.cc:492] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: Bootstrap starting.
I20260926 19:43:46.731570 20297 tablet_bootstrap.cc:654] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:46.732779 20297 log.cc:824] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:46.733148 20298 tablet_bootstrap.cc:654] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:46.733143 20299 tablet_bootstrap.cc:492] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5: Bootstrap starting.
I20260926 19:43:46.734512 20298 log.cc:824] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:46.735894 20299 tablet_bootstrap.cc:654] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:46.736763 20299 log.cc:824] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:46.739540 20298 tablet_bootstrap.cc:492] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: No bootstrap required, opened a new log
I20260926 19:43:46.739588 20299 tablet_bootstrap.cc:492] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5: No bootstrap required, opened a new log
I20260926 19:43:46.739780 20298 ts_tablet_manager.cc:1397] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 19:43:46.739830 20299 ts_tablet_manager.cc:1397] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 19:43:46.740402 20297 tablet_bootstrap.cc:492] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: No bootstrap required, opened a new log
I20260926 19:43:46.740653 20297 ts_tablet_manager.cc:1397] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260926 19:43:46.747026 20298 raft_consensus.cc:348] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41413 } } peers { permanent_uuid: "3ced521f5cf840289794815022d2d3e5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43797 } } peers { permanent_uuid: "e3182c88e6df49d09167f902afb72f5e" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44313 } }
I20260926 19:43:46.747345 20298 raft_consensus.cc:374] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:46.747454 20298 raft_consensus.cc:729] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3182c88e6df49d09167f902afb72f5e, State: Initialized, Role: FOLLOWER
I20260926 19:43:46.747991 20298 consensus_queue.cc:260] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41413 } } peers { permanent_uuid: "3ced521f5cf840289794815022d2d3e5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43797 } } peers { permanent_uuid: "e3182c88e6df49d09167f902afb72f5e" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44313 } }
I20260926 19:43:46.750039 20299 raft_consensus.cc:348] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41413 } } peers { permanent_uuid: "3ced521f5cf840289794815022d2d3e5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43797 } } peers { permanent_uuid: "e3182c88e6df49d09167f902afb72f5e" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44313 } }
I20260926 19:43:46.750509 20299 raft_consensus.cc:374] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:46.750609 20299 raft_consensus.cc:729] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ced521f5cf840289794815022d2d3e5, State: Initialized, Role: FOLLOWER
I20260926 19:43:46.751173 20299 consensus_queue.cc:260] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41413 } } peers { permanent_uuid: "3ced521f5cf840289794815022d2d3e5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43797 } } peers { permanent_uuid: "e3182c88e6df49d09167f902afb72f5e" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44313 } }
I20260926 19:43:46.752893 20299 ts_tablet_manager.cc:1428] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260926 19:43:46.751928 20297 raft_consensus.cc:348] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41413 } } peers { permanent_uuid: "3ced521f5cf840289794815022d2d3e5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43797 } } peers { permanent_uuid: "e3182c88e6df49d09167f902afb72f5e" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44313 } }
I20260926 19:43:46.753558 20297 raft_consensus.cc:374] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:46.753698 20297 raft_consensus.cc:729] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6f44fafb784c6bb0b069c10d2b2c8d, State: Initialized, Role: FOLLOWER
I20260926 19:43:46.753957 20279 heartbeater.cc:503] Master 127.2.164.126:36087 was elected leader, sending a full tablet report...
I20260926 19:43:46.754280 20297 consensus_queue.cc:260] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41413 } } peers { permanent_uuid: "3ced521f5cf840289794815022d2d3e5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43797 } } peers { permanent_uuid: "e3182c88e6df49d09167f902afb72f5e" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44313 } }
I20260926 19:43:46.754534 20298 ts_tablet_manager.cc:1428] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260926 19:43:46.755584 20146 heartbeater.cc:503] Master 127.2.164.126:36087 was elected leader, sending a full tablet report...
I20260926 19:43:46.756168 20297 ts_tablet_manager.cc:1428] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260926 19:43:46.875598 20280 tablet.cc:2369] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:43:46.899869 20303 raft_consensus.cc:482] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:43:46.900135 20303 raft_consensus.cc:504] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41413 } } peers { permanent_uuid: "3ced521f5cf840289794815022d2d3e5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43797 } } peers { permanent_uuid: "e3182c88e6df49d09167f902afb72f5e" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44313 } }
I20260926 19:43:46.901743 20303 leader_election.cc:288] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6f44fafb784c6bb0b069c10d2b2c8d (127.2.164.66:41413), 3ced521f5cf840289794815022d2d3e5 (127.2.164.65:43797)
W20260926 19:43:46.906567 20147 tablet.cc:2369] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:43:46.907961 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.902199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:57533 (local address 127.2.164.65:43797)
0926 19:43:46.902402 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:46.902408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:46.902590 (+ 182us) server_negotiation.cc:407] Connection header received
0926 19:43:46.902797 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:46.902802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:46.902883 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:46.902989 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:46.903454 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.904239 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.906051 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.907158 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.907292 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.907367 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:46.907597 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 19:43:46.907717 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:43:46.908038 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.902112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43797 (local address 127.2.164.67:57533)
0926 19:43:46.902441 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:46.902489 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:46.902646 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:46.903018 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:46.903023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:46.903052 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:46.903262 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.903272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.904362 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.904368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.905893 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.905905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.907293 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.907299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.907458 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.907481 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:43:46.907557 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 19:43:46.907691 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":18,"spinlock_wait_cycles":17152}
I20260926 19:43:46.908721 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.902257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:56383 (local address 127.2.164.66:41413)
0926 19:43:46.902436 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:46.902441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:46.903016 (+ 575us) server_negotiation.cc:407] Connection header received
0926 19:43:46.903306 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:46.903311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:46.903388 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:46.903463 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:46.903947 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.904900 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.906944 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.907861 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.907993 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.908071 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:46.908405 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 19:43:46.908501 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:43:46.908784 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.902182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:41413 (local address 127.2.164.67:56383)
0926 19:43:46.902927 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:46.902944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:46.903140 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:46.903485 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:46.903489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:46.903512 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:46.903812 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.903819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.905048 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.905052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.906779 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:46.906794 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.908067 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.908073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:46.908247 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.908279 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 19:43:46.908364 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 19:43:46.908486 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":1,"thread_start_us":405,"threads_started":1}
I20260926 19:43:46.909070 19967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e1c6461d2e4e03b7d25e4580e86cde" candidate_uuid: "e3182c88e6df49d09167f902afb72f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ced521f5cf840289794815022d2d3e5" is_pre_election: true
I20260926 19:43:46.909650 19967 raft_consensus.cc:2454] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3182c88e6df49d09167f902afb72f5e in term 0.
I20260926 19:43:46.909667 20101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e1c6461d2e4e03b7d25e4580e86cde" candidate_uuid: "e3182c88e6df49d09167f902afb72f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" is_pre_election: true
I20260926 19:43:46.910149 20101 raft_consensus.cc:2454] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3182c88e6df49d09167f902afb72f5e in term 0.
I20260926 19:43:46.910310 20167 leader_election.cc:302] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ced521f5cf840289794815022d2d3e5, e3182c88e6df49d09167f902afb72f5e; no voters:
I20260926 19:43:46.910707 20303 raft_consensus.cc:2781] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:43:46.910864 20303 raft_consensus.cc:482] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:43:46.910961 20303 raft_consensus.cc:3037] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:46.913007 20303 raft_consensus.cc:504] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41413 } } peers { permanent_uuid: "3ced521f5cf840289794815022d2d3e5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43797 } } peers { permanent_uuid: "e3182c88e6df49d09167f902afb72f5e" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44313 } }
I20260926 19:43:46.913604 20303 leader_election.cc:288] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [CANDIDATE]: Term 1 election: Requested vote from peers ac6f44fafb784c6bb0b069c10d2b2c8d (127.2.164.66:41413), 3ced521f5cf840289794815022d2d3e5 (127.2.164.65:43797)
I20260926 19:43:46.913916 20101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e1c6461d2e4e03b7d25e4580e86cde" candidate_uuid: "e3182c88e6df49d09167f902afb72f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d"
I20260926 19:43:46.914034 19967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41e1c6461d2e4e03b7d25e4580e86cde" candidate_uuid: "e3182c88e6df49d09167f902afb72f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ced521f5cf840289794815022d2d3e5"
I20260926 19:43:46.914080 20101 raft_consensus.cc:3037] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:46.914222 19967 raft_consensus.cc:3037] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:46.916145 20101 raft_consensus.cc:2454] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3182c88e6df49d09167f902afb72f5e in term 1.
I20260926 19:43:46.916306 19967 raft_consensus.cc:2454] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3182c88e6df49d09167f902afb72f5e in term 1.
I20260926 19:43:46.916487 20170 leader_election.cc:302] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac6f44fafb784c6bb0b069c10d2b2c8d, e3182c88e6df49d09167f902afb72f5e; no voters:
I20260926 19:43:46.916745 20303 raft_consensus.cc:2781] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:43:46.917196 20303 raft_consensus.cc:686] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 1 LEADER]: Becoming Leader. State: Replica: e3182c88e6df49d09167f902afb72f5e, State: Running, Role: LEADER
I20260926 19:43:46.917709 20303 consensus_queue.cc:237] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41413 } } peers { permanent_uuid: "3ced521f5cf840289794815022d2d3e5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43797 } } peers { permanent_uuid: "e3182c88e6df49d09167f902afb72f5e" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44313 } }
I20260926 19:43:46.922204 19822 catalog_manager.cc:5495] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e reported cstate change: term changed from 0 to 1, leader changed from <none> to e3182c88e6df49d09167f902afb72f5e (127.2.164.67). New cstate: current_term: 1 leader_uuid: "e3182c88e6df49d09167f902afb72f5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ced521f5cf840289794815022d2d3e5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3182c88e6df49d09167f902afb72f5e" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 44313 } health_report { overall_health: HEALTHY } } }
W20260926 19:43:46.929001 20013 tablet.cc:2369] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:43:46.941079 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.936125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.2.164.67:44313)
0926 19:43:46.936327 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:46.936334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:46.936391 (+ 57us) server_negotiation.cc:407] Connection header received
0926 19:43:46.936564 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:46.936568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:46.936645 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:46.936768 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:43:46.937079 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.937914 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.939262 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.939626 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.939710 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.940111 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:43:46.940580 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:43:46.940685 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:46.940789 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 19:43:46.940852 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 19:43:46.948797 19967 raft_consensus.cc:1261] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5 [term 1 FOLLOWER]: Refusing update from remote peer e3182c88e6df49d09167f902afb72f5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:46.948818 20101 raft_consensus.cc:1261] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d [term 1 FOLLOWER]: Refusing update from remote peer e3182c88e6df49d09167f902afb72f5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:46.949729 20307 consensus_queue.cc:1035] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ced521f5cf840289794815022d2d3e5" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:43:46.950073 20303 consensus_queue.cc:1035] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6f44fafb784c6bb0b069c10d2b2c8d" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:43:46.960651 20312 mvcc.cc:205] Tried to move back new op lower bound from 7333690683173310464 to 7333690683058892800. Current Snapshot: MvccSnapshot[applied={T|T < 7333690683173310464}]
I20260926 19:43:46.961647 20310 mvcc.cc:205] Tried to move back new op lower bound from 7333690683173310464 to 7333690683058892800. Current Snapshot: MvccSnapshot[applied={T|T < 7333690683173310464}]
I20260926 19:43:46.975924 19821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:46.976182 19821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:46.979753 19821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6166c5d6aff249c5a40a4a46e9eeeeea: Sending DeleteTablet for 3 replicas of tablet 41e1c6461d2e4e03b7d25e4580e86cde
I20260926 19:43:46.980872 20214 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e1c6461d2e4e03b7d25e4580e86cde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:46 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:60980
I20260926 19:43:46.980901 19947 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e1c6461d2e4e03b7d25e4580e86cde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:46 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:36898
I20260926 19:43:46.982061 20081 tablet_service.cc:1514] Processing DeleteTablet for tablet 41e1c6461d2e4e03b7d25e4580e86cde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:46 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:52530
I20260926 19:43:46.983379 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 19884
I20260926 19:43:46.994309 20320 tablet_replica.cc:330] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5: stopping tablet replica
I20260926 19:43:46.994809 20320 raft_consensus.cc:2229] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:46.995378 20320 raft_consensus.cc:2258] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 19:43:46 dist-test-slave-mrw4 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:43:46.996721 20321 tablet_replica.cc:330] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: stopping tablet replica
I20260926 19:43:46.996841 20322 tablet_replica.cc:330] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: stopping tablet replica
I20260926 19:43:46.997187 20322 raft_consensus.cc:2229] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:46.997145 20321 raft_consensus.cc:2229] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:43:46.997675 20322 raft_consensus.cc:2258] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:46.997859 20321 raft_consensus.cc:2258] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:46.999603 20320 ts_tablet_manager.cc:1905] T 41e1c6461d2e4e03b7d25e4580e86cde P 3ced521f5cf840289794815022d2d3e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:47.002806 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:46.984634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.2.164.65:43797)
0926 19:43:46.984824 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:46.984831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:46.985314 (+ 483us) server_negotiation.cc:407] Connection header received
0926 19:43:46.985583 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:46.985589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:46.985696 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:46.985780 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:46.987676 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.988984 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.991107 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:46.991831 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:46.991935 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:46.997087 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:46.997099 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:46.997112 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:46.997143 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:47.000005 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:47.000634 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:47.000641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:47.000643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:47.000791 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:47.001142 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:47.001145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:47.001147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:47.001598 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:47.001897 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:47.002198 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 19:43:47.002413 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 19:43:47.003508 20007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:47.006379 20321 ts_tablet_manager.cc:1905] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:47.006428 20322 ts_tablet_manager.cc:1905] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:47.013103 20322 ts_tablet_manager.cc:1918] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:47.013324 20321 ts_tablet_manager.cc:1918] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:47.013404 20322 log.cc:1196] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/41e1c6461d2e4e03b7d25e4580e86cde
I20260926 19:43:47.013638 20321 log.cc:1196] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/41e1c6461d2e4e03b7d25e4580e86cde
I20260926 19:43:47.014067 20322 ts_tablet_manager.cc:1939] T 41e1c6461d2e4e03b7d25e4580e86cde P ac6f44fafb784c6bb0b069c10d2b2c8d: Deleting consensus metadata
I20260926 19:43:47.014176 20321 ts_tablet_manager.cc:1939] T 41e1c6461d2e4e03b7d25e4580e86cde P e3182c88e6df49d09167f902afb72f5e: Deleting consensus metadata
I20260926 19:43:47.015905 19810 catalog_manager.cc:4841] TS ac6f44fafb784c6bb0b069c10d2b2c8d (127.2.164.66:41413): tablet 41e1c6461d2e4e03b7d25e4580e86cde (table test-table [id=76d96a3620cb43b69fd6d4be132494f9]) successfully deleted
I20260926 19:43:47.016203 19810 catalog_manager.cc:4841] TS e3182c88e6df49d09167f902afb72f5e (127.2.164.67:44313): tablet 41e1c6461d2e4e03b7d25e4580e86cde (table test-table [id=76d96a3620cb43b69fd6d4be132494f9]) successfully deleted
I20260926 19:43:47.117628 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 19884
I20260926 19:43:47.139159 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 20018
W20260926 19:43:47.139122 19807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:43797 (error 108)
I20260926 19:43:47.141637 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:47.140979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43797 (local address 127.0.0.1:35658)
0926 19:43:47.141337 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:47.141466 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 19:43:47.141881 19807 catalog_manager.cc:4584] TS 3ced521f5cf840289794815022d2d3e5 (127.2.164.65:43797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41e1c6461d2e4e03b7d25e4580e86cde: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43797: connect: Connection refused (error 111)
Sep 26 19:43:47 dist-test-slave-mrw4 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:43:47.154299 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:47.140675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.2.164.66:41413)
0926 19:43:47.140875 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:47.140881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:47.140899 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:43:47.140962 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:47.140967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:47.141043 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:47.141138 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:47.142522 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:47.143222 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:47.144493 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:47.144875 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:47.144979 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:47.149007 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:47.149020 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:47.149035 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:47.149067 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:47.151637 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:47.152266 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:47.152273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:47.152276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:47.152394 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:47.152743 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:47.152746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:47.152748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:47.153181 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:47.153494 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:47.153765 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 19:43:47.153966 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:43:47.155057 20141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:47.195655 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:47.195119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43797 (local address 127.0.0.1:35666)
0926 19:43:47.195359 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:47.195465 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 19:43:47.196830 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:47.196410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43797 (local address 127.0.0.1:35676)
0926 19:43:47.196578 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:47.196668 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 19:43:47.242878 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 20018
I20260926 19:43:47.257123 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 20151
I20260926 19:43:47.257714 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:47.257209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43797 (local address 127.0.0.1:35686)
0926 19:43:47.257425 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:47.257531 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 19:43:47.259063 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:47.258651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43797 (local address 127.0.0.1:35700)
0926 19:43:47.258834 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:47.258923 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
Sep 26 19:43:47 dist-test-slave-mrw4 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:43:47.273926 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:47.258110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.2.164.67:44313)
0926 19:43:47.258316 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:47.258321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:47.258337 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:43:47.258397 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:47.258402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:47.258471 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:47.258585 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:47.260063 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:47.260886 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:47.262163 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:47.262601 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:47.262706 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:47.266997 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:47.267012 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:47.267033 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:47.267088 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:47.270926 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:47.271611 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:47.271645 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:47.271649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:47.271910 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:47.272273 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:47.272279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:47.272282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:47.272789 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:47.273142 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:47.273323 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 19:43:47.273542 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:43:47.274631 20274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:47.339730 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:47.339249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43797 (local address 127.0.0.1:35706)
0926 19:43:47.339463 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:47.339557 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 19:43:47.340739 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:47.340374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43797 (local address 127.0.0.1:35720)
0926 19:43:47.340560 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:47.340629 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 19:43:47.392316 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 20151
I20260926 19:43:47.407537 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 19791
I20260926 19:43:47.408594 19853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:47.519218 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:47.518777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43797 (local address 127.0.0.1:35730)
0926 19:43:47.518966 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:47.519056 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 19:43:47.519733 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 19791
2026-09-26T19:43:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4021 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T19:43:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:43:47Z Disabled control of system clock
I20260926 19:43:47.585570 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34269
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:34269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:43:47.892515 20332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:47.892771 20332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:47.892874 20332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:47.899741 20332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:43:47.899829 20332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:47.899892 20332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:47.899959 20332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:43:47.900015 20332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:43:47.908679 20332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:34269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:34269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:47.909832 20332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:47.911521 20332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:47.922276 20337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:47.922238 20338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:47.922317 20340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:47.923138 20332 server_base.cc:1034] running on GCE node
I20260926 19:43:47.923868 20332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:47.925194 20332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:47.926419 20332 hybrid_clock.cc:648] HybridClock initialized: now 1790451827926368 us; error 46 us; skew 500 ppm
I20260926 19:43:47.926766 20332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:43:47.928869 20332 webserver.cc:460] Webserver started at http://127.2.164.126:46189/ using document root <none> and password file <none>
I20260926 19:43:47.929531 20332 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:47.929628 20332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:47.929915 20332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:47.932580 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "83db37c26fe0498cb5cb2f8ff8e309aa"
format_stamp: "Formatted at 2026-09-26 19:43:47 on dist-test-slave-mrw4"
server_key: "685288e3c839e7e3b5940401faf2dacc"
server_key_iv: "8ce338a29cd694b58ebf837e1be6ed65"
server_key_version: "encryptionkey@0"
I20260926 19:43:47.933369 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "83db37c26fe0498cb5cb2f8ff8e309aa"
format_stamp: "Formatted at 2026-09-26 19:43:47 on dist-test-slave-mrw4"
server_key: "685288e3c839e7e3b5940401faf2dacc"
server_key_iv: "8ce338a29cd694b58ebf837e1be6ed65"
server_key_version: "encryptionkey@0"
I20260926 19:43:47.938076 20332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:43:47.941226 20346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:47.942389 20332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:43:47.942535 20332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "83db37c26fe0498cb5cb2f8ff8e309aa"
format_stamp: "Formatted at 2026-09-26 19:43:47 on dist-test-slave-mrw4"
server_key: "685288e3c839e7e3b5940401faf2dacc"
server_key_iv: "8ce338a29cd694b58ebf837e1be6ed65"
server_key_version: "encryptionkey@0"
I20260926 19:43:47.942727 20332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:47.966339 20332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:47.967134 20332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:47.967388 20332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:47.984097 20332 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:34269
I20260926 19:43:47.984105 20397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:34269 every 8 connection(s)
I20260926 19:43:47.985395 20332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:43:47.989064 20398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:47.994127 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 20332
I20260926 19:43:47.994282 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:43:47.994735 2705 external_mini_cluster.cc:1428] Setting key 4278a2c9e213cdc99fbe2e2bd0d8f0e6
I20260926 19:43:47.999380 20398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa: Bootstrap starting.
I20260926 19:43:48.002774 20398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:48.004020 20398 log.cc:824] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:48.007087 20398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa: No bootstrap required, opened a new log
I20260926 19:43:48.008529 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:47.996648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.2.164.126:34269)
0926 19:43:47.997774 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:47.997784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:47.997823 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:43:47.999444 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:47.999464 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:47.999938 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:48.000296 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:48.000846 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:48.002186 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:48.004084 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:48.005072 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:48.005304 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:48.006361 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:48.006368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:48.006381 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:48.006889 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:48.006961 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:48.006982 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:48.007069 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:48.007317 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 19:43:48.007621 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":348,"threads_started":1}
I20260926 19:43:48.014389 20398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83db37c26fe0498cb5cb2f8ff8e309aa" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34269 } }
I20260926 19:43:48.014707 20398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:48.014794 20398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83db37c26fe0498cb5cb2f8ff8e309aa, State: Initialized, Role: FOLLOWER
I20260926 19:43:48.015300 20398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83db37c26fe0498cb5cb2f8ff8e309aa" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34269 } }
I20260926 19:43:48.015516 20398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:43:48.015632 20398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:43:48.015774 20398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:48.018150 20398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83db37c26fe0498cb5cb2f8ff8e309aa" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34269 } }
I20260926 19:43:48.018584 20398 leader_election.cc:302] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83db37c26fe0498cb5cb2f8ff8e309aa; no voters:
I20260926 19:43:48.019098 20398 leader_election.cc:288] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:43:48.019230 20403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:43:48.019912 20403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [term 1 LEADER]: Becoming Leader. State: Replica: 83db37c26fe0498cb5cb2f8ff8e309aa, State: Running, Role: LEADER
I20260926 19:43:48.020403 20403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83db37c26fe0498cb5cb2f8ff8e309aa" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34269 } }
I20260926 19:43:48.020951 20398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:43:48.023161 20405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83db37c26fe0498cb5cb2f8ff8e309aa. Latest consensus state: current_term: 1 leader_uuid: "83db37c26fe0498cb5cb2f8ff8e309aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83db37c26fe0498cb5cb2f8ff8e309aa" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34269 } } }
I20260926 19:43:48.023135 20404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83db37c26fe0498cb5cb2f8ff8e309aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83db37c26fe0498cb5cb2f8ff8e309aa" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 34269 } } }
I20260926 19:43:48.024979 20404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:48.025513 20405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:48.025694 20413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:43:48.030273 20413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:43:48.038671 20413 catalog_manager.cc:1347] Generated new cluster ID: 2e4426c79d39419fa8d4c7e79bc3626e
I20260926 19:43:48.038776 20413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:43:48.058108 20413 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:43:48.059309 20413 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:43:48.065294 20413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa: Generated new TSK 0
I20260926 19:43:48.066190 20413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:43:48.079928 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34269
--builtin_ntp_servers=127.2.164.84:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:43:48.389802 20422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:48.390054 20422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:48.390177 20422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:48.397451 20422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:48.397549 20422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:48.397684 20422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:43:48.406601 20422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:48.407775 20422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:48.409564 20422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:48.421638 20427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:48.421662 20430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:48.421694 20428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:48.422441 20422 server_base.cc:1034] running on GCE node
I20260926 19:43:48.422995 20422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:48.423856 20422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:48.425051 20422 hybrid_clock.cc:648] HybridClock initialized: now 1790451828425008 us; error 34 us; skew 500 ppm
I20260926 19:43:48.425369 20422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:43:48.432827 20422 webserver.cc:460] Webserver started at http://127.2.164.65:32935/ using document root <none> and password file <none>
I20260926 19:43:48.433504 20422 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:48.433593 20422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:48.433890 20422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:48.436522 20422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "d4977f161e2b4fd2866b389e481ec594"
format_stamp: "Formatted at 2026-09-26 19:43:48 on dist-test-slave-mrw4"
server_key: "b7a9229986f94767e4b3b02c46ace887"
server_key_iv: "306680bdc6565079f43200356c1c00b1"
server_key_version: "encryptionkey@0"
I20260926 19:43:48.437278 20422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "d4977f161e2b4fd2866b389e481ec594"
format_stamp: "Formatted at 2026-09-26 19:43:48 on dist-test-slave-mrw4"
server_key: "b7a9229986f94767e4b3b02c46ace887"
server_key_iv: "306680bdc6565079f43200356c1c00b1"
server_key_version: "encryptionkey@0"
I20260926 19:43:48.442015 20422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260926 19:43:48.445187 20436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:48.446496 20422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:43:48.446636 20422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "d4977f161e2b4fd2866b389e481ec594"
format_stamp: "Formatted at 2026-09-26 19:43:48 on dist-test-slave-mrw4"
server_key: "b7a9229986f94767e4b3b02c46ace887"
server_key_iv: "306680bdc6565079f43200356c1c00b1"
server_key_version: "encryptionkey@0"
I20260926 19:43:48.446830 20422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:48.461268 20422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:48.461980 20422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:48.462226 20422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:48.463073 20422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:48.464370 20422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:48.464452 20422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:48.464551 20422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:48.464622 20422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:48.493113 20422 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:41211
I20260926 19:43:48.493186 20548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:41211 every 8 connection(s)
I20260926 19:43:48.494323 20422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:43:48.500542 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 20422
I20260926 19:43:48.500705 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:43:48.501130 2705 external_mini_cluster.cc:1428] Setting key 9d8308b3acd36d4dce999a066c86c2ad
I20260926 19:43:48.504017 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34269
--builtin_ntp_servers=127.2.164.84:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:43:48.506515 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:48.496720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:36303 (local address 127.2.164.126:34269)
0926 19:43:48.496895 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:48.496900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:48.498209 (+ 1309us) server_negotiation.cc:407] Connection header received
0926 19:43:48.499950 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:48.499955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:48.500033 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:48.500118 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:48.501216 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:48.502018 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:48.503884 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:48.504336 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:48.504441 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:48.505768 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:48.505774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:48.505779 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:48.505798 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:48.505829 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:48.505841 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:48.505914 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:48.506115 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 19:43:48.506213 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:43:48.507495 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:48.497240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34269 (local address 127.2.164.65:36303)
0926 19:43:48.498058 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:48.498097 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:48.499688 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:48.500369 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:48.500378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:48.500464 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:48.501041 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:48.501059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:48.502166 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:48.502170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:48.503747 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:48.503761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:48.504541 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:48.504549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:48.504758 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:48.505425 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:43:48.505429 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:48.505482 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:43:48.505583 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:48.505923 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:48.505934 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:48.505942 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 19:43:48.506094 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 19:43:48.506497 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":245,"threads_started":1}
I20260926 19:43:48.509114 20549 heartbeater.cc:347] Connected to a master server at 127.2.164.126:34269
I20260926 19:43:48.509410 20549 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:48.510171 20549 heartbeater.cc:511] Master 127.2.164.126:34269 requested a full tablet report, sending...
I20260926 19:43:48.512368 20363 ts_manager.cc:194] Registered new tserver with Master: d4977f161e2b4fd2866b389e481ec594 (127.2.164.65:41211)
I20260926 19:43:48.514343 20363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.65:36303
W20260926 19:43:48.811586 20552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:48.811837 20552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:48.811918 20552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:48.818797 20552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:48.818879 20552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:48.818987 20552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:43:48.827494 20552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:48.828586 20552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:48.830258 20552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:48.842180 20557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:48.842236 20558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:48.842249 20560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:48.843017 20552 server_base.cc:1034] running on GCE node
I20260926 19:43:48.843545 20552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:48.844372 20552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:48.845567 20552 hybrid_clock.cc:648] HybridClock initialized: now 1790451828845529 us; error 27 us; skew 500 ppm
I20260926 19:43:48.845851 20552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:43:48.847977 20552 webserver.cc:460] Webserver started at http://127.2.164.66:40655/ using document root <none> and password file <none>
I20260926 19:43:48.848639 20552 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:48.848762 20552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:48.849109 20552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:48.851866 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "cd6ba8c24174457b8d70e667918818a3"
format_stamp: "Formatted at 2026-09-26 19:43:48 on dist-test-slave-mrw4"
server_key: "04a1ad00739a0785beef67d621d38f8f"
server_key_iv: "b3b30685d59c9449cc691229de829736"
server_key_version: "encryptionkey@0"
I20260926 19:43:48.852562 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "cd6ba8c24174457b8d70e667918818a3"
format_stamp: "Formatted at 2026-09-26 19:43:48 on dist-test-slave-mrw4"
server_key: "04a1ad00739a0785beef67d621d38f8f"
server_key_iv: "b3b30685d59c9449cc691229de829736"
server_key_version: "encryptionkey@0"
I20260926 19:43:48.857237 20552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 19:43:48.860306 20566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:48.861568 20552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:43:48.861699 20552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "cd6ba8c24174457b8d70e667918818a3"
format_stamp: "Formatted at 2026-09-26 19:43:48 on dist-test-slave-mrw4"
server_key: "04a1ad00739a0785beef67d621d38f8f"
server_key_iv: "b3b30685d59c9449cc691229de829736"
server_key_version: "encryptionkey@0"
I20260926 19:43:48.861876 20552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:48.872212 20552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:48.872876 20552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:48.873117 20552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:48.873996 20552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:48.875310 20552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:48.875391 20552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:48.875490 20552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:48.875553 20552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:48.903776 20552 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:44033
I20260926 19:43:48.903816 20678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:44033 every 8 connection(s)
I20260926 19:43:48.904981 20552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:43:48.906570 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 20552
I20260926 19:43:48.906738 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:43:48.907148 2705 external_mini_cluster.cc:1428] Setting key 2e8b872a59b02daf94c54dfc0bf9a5a5
I20260926 19:43:48.909916 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34269
--builtin_ntp_servers=127.2.164.84:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:43:48.917126 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:48.907732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:60491 (local address 127.2.164.126:34269)
0926 19:43:48.907903 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:48.907908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:48.909206 (+ 1298us) server_negotiation.cc:407] Connection header received
0926 19:43:48.910924 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:48.910931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:48.911004 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:48.911098 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:48.912065 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:48.912956 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:48.914463 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:48.915015 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:48.915126 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:48.916355 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:48.916361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:48.916366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:48.916387 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:48.916425 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:48.916438 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:48.916521 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:48.916763 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 19:43:48.916858 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:43:48.918390 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:48.907962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34269 (local address 127.2.164.66:60491)
0926 19:43:48.909023 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:48.909068 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:48.910672 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:48.911298 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:48.911307 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:48.911396 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:48.911924 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:48.911939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:48.913070 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:48.913074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:48.914328 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:48.914339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:48.915222 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:48.915227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:48.915417 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:48.916054 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:43:48.916057 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:48.916100 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:43:48.916169 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:48.916562 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:48.916572 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:48.916579 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 19:43:48.916741 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 19:43:48.917078 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":290,"threads_started":1}
I20260926 19:43:48.919680 20679 heartbeater.cc:347] Connected to a master server at 127.2.164.126:34269
I20260926 19:43:48.919960 20679 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:48.920677 20679 heartbeater.cc:511] Master 127.2.164.126:34269 requested a full tablet report, sending...
I20260926 19:43:48.922127 20363 ts_manager.cc:194] Registered new tserver with Master: cd6ba8c24174457b8d70e667918818a3 (127.2.164.66:44033)
I20260926 19:43:48.922902 20363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.66:60491
W20260926 19:43:49.220842 20682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:49.221117 20682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:49.221257 20682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:49.228367 20682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:49.228462 20682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:49.228603 20682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:43:49.237133 20682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:34269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:49.238338 20682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:49.239993 20682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:49.252013 20687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:49.252137 20690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:49.252163 20688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:49.252944 20682 server_base.cc:1034] running on GCE node
I20260926 19:43:49.253516 20682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:49.254366 20682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:49.255568 20682 hybrid_clock.cc:648] HybridClock initialized: now 1790451829255513 us; error 44 us; skew 500 ppm
I20260926 19:43:49.255851 20682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:43:49.258001 20682 webserver.cc:460] Webserver started at http://127.2.164.67:35109/ using document root <none> and password file <none>
I20260926 19:43:49.258631 20682 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:49.258713 20682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:49.258963 20682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:49.261637 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2"
format_stamp: "Formatted at 2026-09-26 19:43:49 on dist-test-slave-mrw4"
server_key: "b15993fad32f025760a342729f14eca6"
server_key_iv: "08337c3807513a8f993fdc7c39970656"
server_key_version: "encryptionkey@0"
I20260926 19:43:49.262360 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2"
format_stamp: "Formatted at 2026-09-26 19:43:49 on dist-test-slave-mrw4"
server_key: "b15993fad32f025760a342729f14eca6"
server_key_iv: "08337c3807513a8f993fdc7c39970656"
server_key_version: "encryptionkey@0"
I20260926 19:43:49.267110 20682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:43:49.270291 20696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:49.271574 20682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:43:49.271718 20682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2"
format_stamp: "Formatted at 2026-09-26 19:43:49 on dist-test-slave-mrw4"
server_key: "b15993fad32f025760a342729f14eca6"
server_key_iv: "08337c3807513a8f993fdc7c39970656"
server_key_version: "encryptionkey@0"
I20260926 19:43:49.271898 20682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:49.287441 20682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:49.288193 20682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:49.288465 20682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:49.289363 20682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:49.290671 20682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:49.290771 20682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:49.290867 20682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:49.290934 20682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:49.319625 20682 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:38527
I20260926 19:43:49.319645 20808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:38527 every 8 connection(s)
I20260926 19:43:49.320812 20682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:43:49.323254 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 20682
I20260926 19:43:49.323411 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:43:49.323843 2705 external_mini_cluster.cc:1428] Setting key 9b73b9d0f905287d4a896858b53ec68c
I20260926 19:43:49.332952 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.323222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:40285 (local address 127.2.164.126:34269)
0926 19:43:49.323431 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:49.323437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:49.324930 (+ 1493us) server_negotiation.cc:407] Connection header received
0926 19:43:49.327028 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:49.327034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:49.327117 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:49.327199 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:49.328246 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.328951 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.330481 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.330885 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.330992 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.332198 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:49.332203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:49.332206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:49.332223 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:49.332253 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:49.332264 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:49.332343 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:49.332543 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 19:43:49.332650 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:43:49.333896 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.323763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:34269 (local address 127.2.164.67:40285)
0926 19:43:49.324738 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.324828 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:49.326790 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:49.327433 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:49.327441 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:49.327528 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:49.328067 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.328083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.329100 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.329103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.330339 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.330351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.331083 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.331090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.331307 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.331921 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:43:49.331924 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:49.331961 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:43:49.332031 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:49.332353 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:49.332363 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:49.332371 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 19:43:49.332520 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 19:43:49.332899 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":287,"threads_started":1}
I20260926 19:43:49.335099 20809 heartbeater.cc:347] Connected to a master server at 127.2.164.126:34269
I20260926 19:43:49.335343 20809 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:49.336010 20809 heartbeater.cc:511] Master 127.2.164.126:34269 requested a full tablet report, sending...
I20260926 19:43:49.337282 20363 ts_manager.cc:194] Registered new tserver with Master: fc3d2f58f457427dae2ac95fe9bc7bb2 (127.2.164.67:38527)
I20260926 19:43:49.338124 20363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.67:40285
I20260926 19:43:49.338852 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:43:49.348008 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.342704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.2.164.126:34269)
0926 19:43:49.342854 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:49.342859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:49.343303 (+ 444us) server_negotiation.cc:407] Connection header received
0926 19:43:49.343424 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:49.343428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:49.343518 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:49.343595 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:49.343970 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.344678 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.346153 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.346532 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.346619 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.347514 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:49.347519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:49.347522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:49.347539 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:49.347565 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:49.347577 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:49.347641 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:49.347716 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 19:43:49.347782 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:43:49.351397 20362 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:43:49.358168 20362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:43:49.380453 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.369355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.2.164.66:44033)
0926 19:43:49.370150 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:49.370159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:49.371292 (+ 1133us) server_negotiation.cc:407] Connection header received
0926 19:43:49.371618 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:49.371626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:49.371871 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:49.372030 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:49.372946 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.374547 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.377447 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.378869 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.378973 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.379245 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:49.379899 (+ 654us) server_negotiation.cc:299] Negotiation successful
0926 19:43:49.380068 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":761,"threads_started":1}
I20260926 19:43:49.381233 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.369152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:44033 (local address 127.0.0.1:54860)
0926 19:43:49.371132 (+ 1980us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.371192 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:49.371380 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:49.372358 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:49.372367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:49.372393 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:49.372756 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.372770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.374714 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.374718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.377225 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.377244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.379405 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.379411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.379612 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.379661 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 19:43:49.380716 (+ 1055us) client_negotiation.cc:240] Negotiation successful
0926 19:43:49.380915 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1663,"thread_start_us":1589,"threads_started":1}
I20260926 19:43:49.382535 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.369224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41211 (local address 127.0.0.1:40096)
0926 19:43:49.371132 (+ 1908us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.371191 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:49.371380 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:49.373358 (+ 1978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:49.373363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:49.373387 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:49.373728 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.373739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.376725 (+ 2986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.376730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.378844 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.378859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.381881 (+ 3022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.381886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.382006 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.382027 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 19:43:49.382090 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 19:43:49.382206 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1714,"mutex_wait_us":1,"thread_start_us":1711,"threads_started":1}
I20260926 19:43:49.382632 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.369761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.2.164.65:41211)
0926 19:43:49.370549 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:49.370558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:49.371411 (+ 853us) server_negotiation.cc:407] Connection header received
0926 19:43:49.371618 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:49.371626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:49.372916 (+ 1290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:49.373099 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:49.373919 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.376573 (+ 2654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.379050 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.381464 (+ 2414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.381578 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.381780 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:49.382176 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 19:43:49.382346 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":521,"threads_started":1}
I20260926 19:43:49.386641 20484 tablet_service.cc:1467] Processing CreateTablet for tablet 4e097d2ec150475ca7ae57367aef06c7 (DEFAULT_TABLE table=test-table [id=b2436594e7344a56962e6daf6a5e7c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:49.387914 20484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e097d2ec150475ca7ae57367aef06c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:49.388988 20614 tablet_service.cc:1467] Processing CreateTablet for tablet 4e097d2ec150475ca7ae57367aef06c7 (DEFAULT_TABLE table=test-table [id=b2436594e7344a56962e6daf6a5e7c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:49.390183 20614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e097d2ec150475ca7ae57367aef06c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:49.392066 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.371557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:38527 (local address 127.0.0.1:50168)
0926 19:43:49.381448 (+ 9891us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.381467 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:49.381559 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:49.386016 (+ 4457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:49.386021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:49.386046 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:49.386185 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.386194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.387899 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.387904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.389651 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.389665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.391372 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.391379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.391551 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.391585 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:43:49.391673 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 19:43:49.391805 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9787,"thread_start_us":14233,"threads_started":1}
I20260926 19:43:49.392251 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.371878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.2.164.67:38527)
0926 19:43:49.385409 (+ 13531us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:49.385418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:49.385454 (+ 36us) server_negotiation.cc:407] Connection header received
0926 19:43:49.385555 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:49.385563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:49.385809 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:49.385990 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:49.386313 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.387753 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.389822 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.391116 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.391255 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.391508 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:49.391743 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 19:43:49.391895 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13329,"thread_start_us":13614,"threads_started":1}
I20260926 19:43:49.395658 20744 tablet_service.cc:1467] Processing CreateTablet for tablet 4e097d2ec150475ca7ae57367aef06c7 (DEFAULT_TABLE table=test-table [id=b2436594e7344a56962e6daf6a5e7c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:49.396996 20744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e097d2ec150475ca7ae57367aef06c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:49.400836 20827 tablet_bootstrap.cc:492] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3: Bootstrap starting.
I20260926 19:43:49.402673 20828 tablet_bootstrap.cc:492] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594: Bootstrap starting.
I20260926 19:43:49.404384 20827 tablet_bootstrap.cc:654] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:49.405576 20827 log.cc:824] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:49.405887 20828 tablet_bootstrap.cc:654] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:49.407061 20828 log.cc:824] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:49.409348 20827 tablet_bootstrap.cc:492] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3: No bootstrap required, opened a new log
I20260926 19:43:49.409360 20831 tablet_bootstrap.cc:492] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2: Bootstrap starting.
I20260926 19:43:49.409605 20827 ts_tablet_manager.cc:1397] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 19:43:49.409960 20828 tablet_bootstrap.cc:492] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594: No bootstrap required, opened a new log
I20260926 19:43:49.410179 20828 ts_tablet_manager.cc:1397] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 19:43:49.412726 20831 tablet_bootstrap.cc:654] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:49.413818 20831 log.cc:824] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:49.416510 20831 tablet_bootstrap.cc:492] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2: No bootstrap required, opened a new log
I20260926 19:43:49.416740 20831 ts_tablet_manager.cc:1397] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 19:43:49.417155 20828 raft_consensus.cc:348] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6ba8c24174457b8d70e667918818a3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44033 } } peers { permanent_uuid: "d4977f161e2b4fd2866b389e481ec594" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41211 } } peers { permanent_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 38527 } }
I20260926 19:43:49.417871 20828 raft_consensus.cc:374] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:49.417961 20828 raft_consensus.cc:729] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4977f161e2b4fd2866b389e481ec594, State: Initialized, Role: FOLLOWER
I20260926 19:43:49.418504 20828 consensus_queue.cc:260] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6ba8c24174457b8d70e667918818a3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44033 } } peers { permanent_uuid: "d4977f161e2b4fd2866b389e481ec594" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41211 } } peers { permanent_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 38527 } }
I20260926 19:43:49.418787 20827 raft_consensus.cc:348] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6ba8c24174457b8d70e667918818a3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44033 } } peers { permanent_uuid: "d4977f161e2b4fd2866b389e481ec594" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41211 } } peers { permanent_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 38527 } }
I20260926 19:43:49.419186 20827 raft_consensus.cc:374] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:49.419310 20827 raft_consensus.cc:729] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6ba8c24174457b8d70e667918818a3, State: Initialized, Role: FOLLOWER
I20260926 19:43:49.419577 20549 heartbeater.cc:503] Master 127.2.164.126:34269 was elected leader, sending a full tablet report...
I20260926 19:43:49.420082 20828 ts_tablet_manager.cc:1428] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260926 19:43:49.419929 20827 consensus_queue.cc:260] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6ba8c24174457b8d70e667918818a3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44033 } } peers { permanent_uuid: "d4977f161e2b4fd2866b389e481ec594" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41211 } } peers { permanent_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 38527 } }
I20260926 19:43:49.421015 20679 heartbeater.cc:503] Master 127.2.164.126:34269 was elected leader, sending a full tablet report...
I20260926 19:43:49.421600 20827 ts_tablet_manager.cc:1428] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 19:43:49.427201 20831 raft_consensus.cc:348] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6ba8c24174457b8d70e667918818a3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44033 } } peers { permanent_uuid: "d4977f161e2b4fd2866b389e481ec594" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41211 } } peers { permanent_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 38527 } }
I20260926 19:43:49.427513 20831 raft_consensus.cc:374] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:49.427601 20831 raft_consensus.cc:729] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3d2f58f457427dae2ac95fe9bc7bb2, State: Initialized, Role: FOLLOWER
I20260926 19:43:49.428045 20831 consensus_queue.cc:260] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6ba8c24174457b8d70e667918818a3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44033 } } peers { permanent_uuid: "d4977f161e2b4fd2866b389e481ec594" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41211 } } peers { permanent_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 38527 } }
I20260926 19:43:49.428900 20809 heartbeater.cc:503] Master 127.2.164.126:34269 was elected leader, sending a full tablet report...
I20260926 19:43:49.429445 20831 ts_tablet_manager.cc:1428] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 19:43:49.435691 20833 raft_consensus.cc:482] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:43:49.435940 20833 raft_consensus.cc:504] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6ba8c24174457b8d70e667918818a3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44033 } } peers { permanent_uuid: "d4977f161e2b4fd2866b389e481ec594" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41211 } } peers { permanent_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 38527 } }
I20260926 19:43:49.437376 20833 leader_election.cc:288] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd6ba8c24174457b8d70e667918818a3 (127.2.164.66:44033), fc3d2f58f457427dae2ac95fe9bc7bb2 (127.2.164.67:38527)
I20260926 19:43:49.444679 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.437686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:42675 (local address 127.2.164.66:44033)
0926 19:43:49.437918 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:49.437925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:49.438301 (+ 376us) server_negotiation.cc:407] Connection header received
0926 19:43:49.438503 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:49.438509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:49.438589 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:49.438700 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:49.439262 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.440215 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.442748 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.443867 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.444043 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.444114 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:49.444355 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 19:43:49.444445 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:43:49.444802 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.437641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:44033 (local address 127.2.164.65:42675)
0926 19:43:49.438197 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.438212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:49.438358 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:49.438725 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:49.438730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:49.438754 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:49.439147 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.439156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.440342 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.440345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.442566 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.442581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.444029 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.444033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.444183 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.444235 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 19:43:49.444336 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 19:43:49.444456 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":120,"thread_start_us":287,"threads_started":1}
I20260926 19:43:49.444859 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.438413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:37021 (local address 127.2.164.67:38527)
0926 19:43:49.438595 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:49.438600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:49.438986 (+ 386us) server_negotiation.cc:407] Connection header received
0926 19:43:49.439129 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:49.439133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:49.439211 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:49.439300 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:49.439720 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.440596 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.442921 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.444050 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.444187 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.444230 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:49.444529 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 19:43:49.444638 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:43:49.444896 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.438315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:38527 (local address 127.2.164.65:37021)
0926 19:43:49.438882 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.438902 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:49.439020 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:49.439323 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:49.439326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:49.439345 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:49.439595 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.439602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.440708 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.440713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.442760 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:49.442773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.444197 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.444203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:49.444383 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.444412 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 19:43:49.444484 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 19:43:49.444589 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":21,"thread_start_us":263,"threads_started":1}
I20260926 19:43:49.446061 20634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e097d2ec150475ca7ae57367aef06c7" candidate_uuid: "d4977f161e2b4fd2866b389e481ec594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6ba8c24174457b8d70e667918818a3" is_pre_election: true
I20260926 19:43:49.446065 20764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e097d2ec150475ca7ae57367aef06c7" candidate_uuid: "d4977f161e2b4fd2866b389e481ec594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" is_pre_election: true
I20260926 19:43:49.446497 20634 raft_consensus.cc:2454] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4977f161e2b4fd2866b389e481ec594 in term 0.
I20260926 19:43:49.446589 20764 raft_consensus.cc:2454] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4977f161e2b4fd2866b389e481ec594 in term 0.
I20260926 19:43:49.447130 20439 leader_election.cc:302] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd6ba8c24174457b8d70e667918818a3, d4977f161e2b4fd2866b389e481ec594; no voters:
I20260926 19:43:49.447476 20833 raft_consensus.cc:2781] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:43:49.447594 20833 raft_consensus.cc:482] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:43:49.447691 20833 raft_consensus.cc:3037] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:49.449877 20833 raft_consensus.cc:504] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6ba8c24174457b8d70e667918818a3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44033 } } peers { permanent_uuid: "d4977f161e2b4fd2866b389e481ec594" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41211 } } peers { permanent_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 38527 } }
I20260926 19:43:49.450488 20833 leader_election.cc:288] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [CANDIDATE]: Term 1 election: Requested vote from peers cd6ba8c24174457b8d70e667918818a3 (127.2.164.66:44033), fc3d2f58f457427dae2ac95fe9bc7bb2 (127.2.164.67:38527)
I20260926 19:43:49.450953 20764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e097d2ec150475ca7ae57367aef06c7" candidate_uuid: "d4977f161e2b4fd2866b389e481ec594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2"
I20260926 19:43:49.450970 20634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e097d2ec150475ca7ae57367aef06c7" candidate_uuid: "d4977f161e2b4fd2866b389e481ec594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6ba8c24174457b8d70e667918818a3"
I20260926 19:43:49.451129 20764 raft_consensus.cc:3037] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:49.451152 20634 raft_consensus.cc:3037] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:49.454078 20764 raft_consensus.cc:2454] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4977f161e2b4fd2866b389e481ec594 in term 1.
I20260926 19:43:49.454306 20634 raft_consensus.cc:2454] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4977f161e2b4fd2866b389e481ec594 in term 1.
I20260926 19:43:49.454447 20439 leader_election.cc:302] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4977f161e2b4fd2866b389e481ec594, fc3d2f58f457427dae2ac95fe9bc7bb2; no voters:
I20260926 19:43:49.455511 20833 raft_consensus.cc:2781] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:43:49.455976 20833 raft_consensus.cc:686] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 1 LEADER]: Becoming Leader. State: Replica: d4977f161e2b4fd2866b389e481ec594, State: Running, Role: LEADER
I20260926 19:43:49.456506 20833 consensus_queue.cc:237] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6ba8c24174457b8d70e667918818a3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44033 } } peers { permanent_uuid: "d4977f161e2b4fd2866b389e481ec594" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41211 } } peers { permanent_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 38527 } }
I20260926 19:43:49.462029 20362 catalog_manager.cc:5495] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4977f161e2b4fd2866b389e481ec594 (127.2.164.65). New cstate: current_term: 1 leader_uuid: "d4977f161e2b4fd2866b389e481ec594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6ba8c24174457b8d70e667918818a3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4977f161e2b4fd2866b389e481ec594" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 38527 } health_report { overall_health: UNKNOWN } } }
I20260926 19:43:49.475636 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.470830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.2.164.65:41211)
0926 19:43:49.470979 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:49.470983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:49.471041 (+ 58us) server_negotiation.cc:407] Connection header received
0926 19:43:49.471202 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:49.471205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:49.471283 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:49.471392 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:43:49.471658 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.472340 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.473745 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.474148 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.474236 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.474657 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:43:49.475166 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:43:49.475262 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:49.475355 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 19:43:49.475415 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 19:43:49.483248 20634 raft_consensus.cc:1261] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3 [term 1 FOLLOWER]: Refusing update from remote peer d4977f161e2b4fd2866b389e481ec594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:49.483246 20764 raft_consensus.cc:1261] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2 [term 1 FOLLOWER]: Refusing update from remote peer d4977f161e2b4fd2866b389e481ec594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:49.484160 20838 consensus_queue.cc:1035] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd6ba8c24174457b8d70e667918818a3" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:43:49.484463 20833 consensus_queue.cc:1035] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc3d2f58f457427dae2ac95fe9bc7bb2" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 38527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:43:49.494789 20841 mvcc.cc:205] Tried to move back new op lower bound from 7333690693554089984 to 7333690693458550784. Current Snapshot: MvccSnapshot[applied={T|T < 7333690693554089984}]
I20260926 19:43:49.500206 20842 mvcc.cc:205] Tried to move back new op lower bound from 7333690693554089984 to 7333690693458550784. Current Snapshot: MvccSnapshot[applied={T|T < 7333690693554089984}]
I20260926 19:43:49.513099 20363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:49.513418 20363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:49.520324 20363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83db37c26fe0498cb5cb2f8ff8e309aa: Sending DeleteTablet for 3 replicas of tablet 4e097d2ec150475ca7ae57367aef06c7
I20260926 19:43:49.521665 20614 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e097d2ec150475ca7ae57367aef06c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:49 UTC) from {username='slave'} at 127.0.0.1:54860
I20260926 19:43:49.521915 20744 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e097d2ec150475ca7ae57367aef06c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:49 UTC) from {username='slave'} at 127.0.0.1:50168
I20260926 19:43:49.522428 20743 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e097d2ec150475ca7ae57367aef06c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:49 UTC) from {username='slave'} at 127.0.0.1:50168
I20260926 19:43:49.522615 20484 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e097d2ec150475ca7ae57367aef06c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:49 UTC) from {username='slave'} at 127.0.0.1:40096
I20260926 19:43:49.523391 20853 tablet_replica.cc:330] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2: stopping tablet replica
I20260926 19:43:49.523782 20853 raft_consensus.cc:2229] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:49.524214 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 20422
I20260926 19:43:49.524217 20853 raft_consensus.cc:2258] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:49.524986 20852 tablet_replica.cc:330] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3: stopping tablet replica
I20260926 19:43:49.525373 20852 raft_consensus.cc:2229] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:49.525918 20852 raft_consensus.cc:2258] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:49.526530 20853 ts_tablet_manager.cc:1905] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:49.531066 20852 ts_tablet_manager.cc:1905] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:49.533315 20853 ts_tablet_manager.cc:1918] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:49.533517 20853 log.cc:1196] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/4e097d2ec150475ca7ae57367aef06c7
I20260926 19:43:49.534045 20853 ts_tablet_manager.cc:1939] T 4e097d2ec150475ca7ae57367aef06c7 P fc3d2f58f457427dae2ac95fe9bc7bb2: Deleting consensus metadata
I20260926 19:43:49.535797 20349 catalog_manager.cc:4841] TS fc3d2f58f457427dae2ac95fe9bc7bb2 (127.2.164.67:38527): tablet 4e097d2ec150475ca7ae57367aef06c7 (table test-table [id=b2436594e7344a56962e6daf6a5e7c83]) successfully deleted
W20260926 19:43:49.536343 20349 catalog_manager.cc:4808] TS fc3d2f58f457427dae2ac95fe9bc7bb2 (127.2.164.67:38527): delete failed for tablet 4e097d2ec150475ca7ae57367aef06c7 because the tablet was not found. No further retry: Not found: Tablet not found: 4e097d2ec150475ca7ae57367aef06c7
I20260926 19:43:49.537774 20852 ts_tablet_manager.cc:1918] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:49.537921 20852 log.cc:1196] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/4e097d2ec150475ca7ae57367aef06c7
I20260926 19:43:49.538417 20852 ts_tablet_manager.cc:1939] T 4e097d2ec150475ca7ae57367aef06c7 P cd6ba8c24174457b8d70e667918818a3: Deleting consensus metadata
I20260926 19:43:49.540073 20349 catalog_manager.cc:4841] TS cd6ba8c24174457b8d70e667918818a3 (127.2.164.66:44033): tablet 4e097d2ec150475ca7ae57367aef06c7 (table test-table [id=b2436594e7344a56962e6daf6a5e7c83]) successfully deleted
I20260926 19:43:49.541676 20854 tablet_replica.cc:330] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594: stopping tablet replica
I20260926 19:43:49.542057 20854 raft_consensus.cc:2229] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:43:49.542920 20854 raft_consensus.cc:2258] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:49.544605 20854 ts_tablet_manager.cc:1905] T 4e097d2ec150475ca7ae57367aef06c7 P d4977f161e2b4fd2866b389e481ec594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:49.545442 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.525958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.2.164.65:41211)
0926 19:43:49.526135 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:49.526142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:49.537620 (+ 11478us) server_negotiation.cc:407] Connection header received
0926 19:43:49.538099 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:49.538109 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:49.538204 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:49.538284 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:49.538977 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.540106 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.541541 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.542234 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.542366 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.543714 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:49.543725 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:49.543740 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:49.544469 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:49.544528 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:49.544561 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:49.544642 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:49.544914 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 19:43:49.545020 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:43:49.546180 20544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:49.639038 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 20422
I20260926 19:43:49.653586 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 20552
W20260926 19:43:49.653900 20347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:41211 (error 108)
I20260926 19:43:49.655975 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.655443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41211 (local address 127.0.0.1:40118)
0926 19:43:49.655647 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.655794 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:41211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 19:43:49.656195 20347 catalog_manager.cc:4584] TS d4977f161e2b4fd2866b389e481ec594 (127.2.164.65:41211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e097d2ec150475ca7ae57367aef06c7: Network error: Client connection negotiation failed: client connection to 127.2.164.65:41211: connect: Connection refused (error 111)
I20260926 19:43:49.660360 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.654324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54864 (local address 127.2.164.66:44033)
0926 19:43:49.654519 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:49.654525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:49.654547 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:43:49.654680 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:49.654684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:49.654784 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:49.654868 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:49.655295 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.656220 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.657621 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.658016 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.658123 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.659124 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:49.659132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:49.659146 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:49.659675 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:49.659723 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:49.659742 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:49.659840 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:49.659964 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 19:43:49.660064 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:43:49.660926 20674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:49.709815 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.709320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41211 (local address 127.0.0.1:40132)
0926 19:43:49.709537 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.709662 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:41211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 19:43:49.710822 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.710480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41211 (local address 127.0.0.1:40146)
0926 19:43:49.710608 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.710677 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:41211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 19:43:49.748960 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 20552
I20260926 19:43:49.762326 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 20682
I20260926 19:43:49.768596 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.763075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.2.164.67:38527)
0926 19:43:49.763233 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:49.763238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:49.763256 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:43:49.763363 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:49.763367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:49.763452 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:49.763535 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:49.763865 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.764607 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.765990 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:49.766382 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:49.766476 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:49.767392 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:49.767401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:49.767416 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:49.768014 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:49.768063 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:49.768083 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:49.768180 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:49.768259 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 19:43:49.768347 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 19:43:49.769408 20804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:49.771209 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.770858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41211 (local address 127.0.0.1:40148)
0926 19:43:49.771009 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.771091 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:41211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 19:43:49.772279 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.771829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41211 (local address 127.0.0.1:40158)
0926 19:43:49.772016 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.772111 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:41211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 19:43:49.852895 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.852378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41211 (local address 127.0.0.1:40166)
0926 19:43:49.852607 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.852718 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:41211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 19:43:49.854486 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:49.854033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:41211 (local address 127.0.0.1:40178)
0926 19:43:49.854229 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:49.854315 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:41211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 19:43:49.855201 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 20682
I20260926 19:43:49.868376 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 20332
I20260926 19:43:49.869469 20393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:49.974865 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 20332
2026-09-26T19:43:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2454 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:43:50 dist-test-slave-mrw4 krb5kdc[20863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:43:50 dist-test-slave-mrw4 krb5kdc[20863](info): set up 2 sockets
Sep 26 19:43:50 dist-test-slave-mrw4 krb5kdc[20863](info): commencing operation
krb5kdc: starting...
W20260926 19:43:50.984937 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:43:51 dist-test-slave-mrw4 krb5kdc[20863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:43:51.204352 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:46623
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:46623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:51.514498 20876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:51.514780 20876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:51.514909 20876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:51.522043 20876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:43:51.522152 20876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:51.522225 20876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:51.522306 20876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:43:51.522367 20876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:43:51.532111 20876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:46623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:46623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:51.533401 20876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:51.534607 20876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:51.545488 20881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:51.545516 20884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:51.545559 20882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:51.546154 20876 server_base.cc:1034] running on GCE node
I20260926 19:43:51.546844 20876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:51.548133 20876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:51.549353 20876 hybrid_clock.cc:648] HybridClock initialized: now 1790451831549294 us; error 33 us; skew 500 ppm
I20260926 19:43:51.549647 20876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:51 dist-test-slave-mrw4 krb5kdc[20863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451831, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:51.552419 20876 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:43:51.553752 20876 webserver.cc:460] Webserver started at http://127.2.164.126:42345/ using document root <none> and password file <none>
I20260926 19:43:51.554358 20876 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:51.554453 20876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:51.554694 20876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:51.557467 20876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "fadb9e37a84847cf8669c024a20773bd"
format_stamp: "Formatted at 2026-09-26 19:43:51 on dist-test-slave-mrw4"
server_key: "c13b81f6ca8d07fa82fed68384c23795"
server_key_iv: "f168da0e68d7a8a63c193ceae875b276"
server_key_version: "encryptionkey@0"
I20260926 19:43:51.558126 20876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "fadb9e37a84847cf8669c024a20773bd"
format_stamp: "Formatted at 2026-09-26 19:43:51 on dist-test-slave-mrw4"
server_key: "c13b81f6ca8d07fa82fed68384c23795"
server_key_iv: "f168da0e68d7a8a63c193ceae875b276"
server_key_version: "encryptionkey@0"
I20260926 19:43:51.562835 20876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:43:51.566011 20891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:51.567217 20876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:43:51.567380 20876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "fadb9e37a84847cf8669c024a20773bd"
format_stamp: "Formatted at 2026-09-26 19:43:51 on dist-test-slave-mrw4"
server_key: "c13b81f6ca8d07fa82fed68384c23795"
server_key_iv: "f168da0e68d7a8a63c193ceae875b276"
server_key_version: "encryptionkey@0"
I20260926 19:43:51.567526 20876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:51.584574 20876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:51.588104 20876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:51.588338 20876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:51.604993 20876 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:46623
I20260926 19:43:51.605016 20942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:46623 every 8 connection(s)
I20260926 19:43:51.606297 20876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:43:51.609850 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:51.614449 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 20876
I20260926 19:43:51.614625 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:43:51.615175 2705 external_mini_cluster.cc:1428] Setting key eb11abdce0a72dd0a8d4fca9aee81dbf
I20260926 19:43:51.620781 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd: Bootstrap starting.
I20260926 19:43:51.624150 20943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:51.625295 20943 log.cc:824] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:51.627853 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd: No bootstrap required, opened a new log
Sep 26 19:43:51 dist-test-slave-mrw4 krb5kdc[20863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:51.635363 20943 raft_consensus.cc:348] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fadb9e37a84847cf8669c024a20773bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46623 } }
I20260926 19:43:51.635748 20943 raft_consensus.cc:374] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:51.635862 20943 raft_consensus.cc:729] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fadb9e37a84847cf8669c024a20773bd, State: Initialized, Role: FOLLOWER
I20260926 19:43:51.636492 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:51.616927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.2.164.126:46623)
0926 19:43:51.617705 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:51.617717 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:51.617767 (+ 50us) server_negotiation.cc:407] Connection header received
0926 19:43:51.619477 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:51.619498 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:51.619846 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:51.620145 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:51.621859 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:51.623166 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:51.624911 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:51.625640 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:51.625859 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:51.629584 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:51.629592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:51.629604 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:51.629655 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:51.632576 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:51.633241 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:51.633249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:51.633258 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:51.633425 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:51.633782 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:51.633786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:51.633788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:51.634296 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:51.634572 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:51.635016 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 19:43:51.635413 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":383,"threads_started":1}
I20260926 19:43:51.636379 20943 consensus_queue.cc:260] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fadb9e37a84847cf8669c024a20773bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46623 } }
I20260926 19:43:51.636653 20943 raft_consensus.cc:388] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:43:51.636812 20943 raft_consensus.cc:482] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:43:51.637009 20943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:51.639539 20943 raft_consensus.cc:504] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fadb9e37a84847cf8669c024a20773bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46623 } }
I20260926 19:43:51.639958 20943 leader_election.cc:302] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fadb9e37a84847cf8669c024a20773bd; no voters:
I20260926 19:43:51.640441 20943 leader_election.cc:288] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:43:51.640554 20948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:43:51.641602 20948 raft_consensus.cc:686] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [term 1 LEADER]: Becoming Leader. State: Replica: fadb9e37a84847cf8669c024a20773bd, State: Running, Role: LEADER
I20260926 19:43:51.642122 20948 consensus_queue.cc:237] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fadb9e37a84847cf8669c024a20773bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46623 } }
I20260926 19:43:51.642370 20943 sys_catalog.cc:564] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:43:51.644721 20950 sys_catalog.cc:455] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fadb9e37a84847cf8669c024a20773bd. Latest consensus state: current_term: 1 leader_uuid: "fadb9e37a84847cf8669c024a20773bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fadb9e37a84847cf8669c024a20773bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46623 } } }
I20260926 19:43:51.645089 20950 sys_catalog.cc:458] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:51.644693 20949 sys_catalog.cc:455] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fadb9e37a84847cf8669c024a20773bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fadb9e37a84847cf8669c024a20773bd" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 46623 } } }
I20260926 19:43:51.645385 20949 sys_catalog.cc:458] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:51.647168 20956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:43:51.651475 20956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:43:51.658818 20956 catalog_manager.cc:1347] Generated new cluster ID: c9b36864c1f14768b303e8528cf33cd2
I20260926 19:43:51.658931 20956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:43:51.677790 20956 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:43:51.679663 20956 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:43:51.689283 20956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd: Generated new TSK 0
I20260926 19:43:51.690068 20956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:43:51.752519 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46623
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:52.063629 20969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:52.063894 20969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:52.064021 20969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:52.071206 20969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:52.071317 20969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:52.071467 20969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:43:52.080932 20969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:52.082158 20969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:52.083395 20969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:52.095425 20974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:52.095397 20975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:52.095371 20977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:52.096515 20969 server_base.cc:1034] running on GCE node
I20260926 19:43:52.097018 20969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:52.097904 20969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:52.099112 20969 hybrid_clock.cc:648] HybridClock initialized: now 1790451832099055 us; error 42 us; skew 500 ppm
I20260926 19:43:52.099401 20969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:52 dist-test-slave-mrw4 krb5kdc[20863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451832, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:52.102300 20969 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:43:52.103592 20969 webserver.cc:460] Webserver started at http://127.2.164.65:35053/ using document root <none> and password file <none>
I20260926 19:43:52.104209 20969 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:52.104321 20969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:52.104616 20969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:52.107348 20969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "0c15f897b12d4b58af05263d514ef015"
format_stamp: "Formatted at 2026-09-26 19:43:52 on dist-test-slave-mrw4"
server_key: "fa66a5fc05d47b1c182546a08cd7a890"
server_key_iv: "1937a2ed18c78ca39cb814540deacc1f"
server_key_version: "encryptionkey@0"
I20260926 19:43:52.108098 20969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "0c15f897b12d4b58af05263d514ef015"
format_stamp: "Formatted at 2026-09-26 19:43:52 on dist-test-slave-mrw4"
server_key: "fa66a5fc05d47b1c182546a08cd7a890"
server_key_iv: "1937a2ed18c78ca39cb814540deacc1f"
server_key_version: "encryptionkey@0"
I20260926 19:43:52.112864 20969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 19:43:52.115969 20984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:52.117213 20969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:43:52.117379 20969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "0c15f897b12d4b58af05263d514ef015"
format_stamp: "Formatted at 2026-09-26 19:43:52 on dist-test-slave-mrw4"
server_key: "fa66a5fc05d47b1c182546a08cd7a890"
server_key_iv: "1937a2ed18c78ca39cb814540deacc1f"
server_key_version: "encryptionkey@0"
I20260926 19:43:52.117544 20969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:52.137638 20969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:52.141148 20969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:52.141438 20969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:52.142310 20969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:52.143544 20969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:52.143620 20969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:52.143738 20969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:52.143796 20969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:52.172052 20969 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:38607
I20260926 19:43:52.172102 21096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:38607 every 8 connection(s)
I20260926 19:43:52.173342 20969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:43:52.183215 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 20969
I20260926 19:43:52.183394 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:43:52.183794 2705 external_mini_cluster.cc:1428] Setting key d04c8fd62ffe5136320f6c8aa6fd82ba
Sep 26 19:43:52 dist-test-slave-mrw4 krb5kdc[20863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451832, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:52.192826 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:52.175699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:48609 (local address 127.2.164.126:46623)
0926 19:43:52.176107 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:52.176111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:52.177070 (+ 959us) server_negotiation.cc:407] Connection header received
0926 19:43:52.178910 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:52.178915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:52.178999 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:52.179091 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:52.180552 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:52.181360 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:52.182796 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:52.183176 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:52.183295 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:52.187012 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:52.187017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:52.187020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:52.187052 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:52.189782 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:52.190614 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:52.190618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:52.190621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:52.190739 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:52.191181 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:52.191186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:52.191197 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:52.191519 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:52.191631 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:52.192411 (+ 780us) server_negotiation.cc:299] Negotiation successful
0926 19:43:52.192571 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":209,"threads_started":1}
I20260926 19:43:52.194103 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:52.176251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:46623 (local address 127.2.164.65:48609)
0926 19:43:52.176918 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:52.176960 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:52.178643 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:52.179343 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:52.179351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:52.179829 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:52.180364 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:52.180382 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:52.181485 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:52.181489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:52.182673 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:52.182684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:52.183415 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:52.183423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:52.183641 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:52.184265 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:52.184304 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:52.186789 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:52.189920 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:52.189929 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:52.189945 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:52.190498 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:52.190833 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:52.190837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:52.190840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:52.191062 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:52.191630 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:52.191638 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:52.192032 (+ 394us) client_negotiation.cc:769] Sending connection context
0926 19:43:52.192324 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 19:43:52.192834 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":257,"threads_started":1}
I20260926 19:43:52.195552 21097 heartbeater.cc:347] Connected to a master server at 127.2.164.126:46623
I20260926 19:43:52.195848 21097 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:52.196640 21097 heartbeater.cc:511] Master 127.2.164.126:46623 requested a full tablet report, sending...
I20260926 19:43:52.198868 20908 ts_manager.cc:194] Registered new tserver with Master: 0c15f897b12d4b58af05263d514ef015 (127.2.164.65:38607)
I20260926 19:43:52.200728 20908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:48609
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:43:52.236039 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46623
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:52.539369 21103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:52.539644 21103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:52.539781 21103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:52.546839 21103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:52.546944 21103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:52.547097 21103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:43:52.556543 21103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:52.557787 21103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:52.559039 21103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:52.571094 21109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:52.571215 21108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:52.571462 21111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:52.572067 21103 server_base.cc:1034] running on GCE node
I20260926 19:43:52.572610 21103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:52.573506 21103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:52.574699 21103 hybrid_clock.cc:648] HybridClock initialized: now 1790451832574653 us; error 41 us; skew 500 ppm
I20260926 19:43:52.574980 21103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:52 dist-test-slave-mrw4 krb5kdc[20863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451832, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:52.577863 21103 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:43:52.579098 21103 webserver.cc:460] Webserver started at http://127.2.164.66:46151/ using document root <none> and password file <none>
I20260926 19:43:52.579694 21103 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:52.579794 21103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:52.580039 21103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:52.582815 21103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "3deb5533ae344dbcab212dcb97460a85"
format_stamp: "Formatted at 2026-09-26 19:43:52 on dist-test-slave-mrw4"
server_key: "276f0ff7ff999ddaf4842ddac406dd03"
server_key_iv: "81edb88aac3531b7797940678b8458c5"
server_key_version: "encryptionkey@0"
I20260926 19:43:52.583483 21103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "3deb5533ae344dbcab212dcb97460a85"
format_stamp: "Formatted at 2026-09-26 19:43:52 on dist-test-slave-mrw4"
server_key: "276f0ff7ff999ddaf4842ddac406dd03"
server_key_iv: "81edb88aac3531b7797940678b8458c5"
server_key_version: "encryptionkey@0"
I20260926 19:43:52.588090 21103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:43:52.591276 21118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:52.592514 21103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:43:52.592649 21103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "3deb5533ae344dbcab212dcb97460a85"
format_stamp: "Formatted at 2026-09-26 19:43:52 on dist-test-slave-mrw4"
server_key: "276f0ff7ff999ddaf4842ddac406dd03"
server_key_iv: "81edb88aac3531b7797940678b8458c5"
server_key_version: "encryptionkey@0"
I20260926 19:43:52.592825 21103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:52.609043 21103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:52.613122 21103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:52.613386 21103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:52.614256 21103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:52.615563 21103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:52.615636 21103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:52.615754 21103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:52.615803 21103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:52.643198 21103 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:37657
I20260926 19:43:52.643240 21230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:37657 every 8 connection(s)
I20260926 19:43:52.644366 21103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:43:52.645589 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 21103
I20260926 19:43:52.645735 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:43:52.646215 2705 external_mini_cluster.cc:1428] Setting key 0d4525ddd5b3b7f0deae07f0ee2cf729
Sep 26 19:43:52 dist-test-slave-mrw4 krb5kdc[20863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451832, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:52.664162 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:52.646995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:42469 (local address 127.2.164.126:46623)
0926 19:43:52.647165 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:52.647169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:52.648507 (+ 1338us) server_negotiation.cc:407] Connection header received
0926 19:43:52.650317 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:52.650323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:52.650400 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:52.650493 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:52.651873 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:52.652836 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:52.654363 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:52.654920 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:52.655037 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:52.658781 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:52.658788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:52.658794 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:52.658833 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:52.661229 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:52.662000 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:52.662004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:52.662006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:52.662128 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:52.662582 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:52.662586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:52.662588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:52.662871 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:52.662991 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:52.663765 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 19:43:52.663937 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:43:52.665438 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:52.647585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:46623 (local address 127.2.164.66:42469)
0926 19:43:52.648348 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:52.648390 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:52.650062 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:52.650684 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:52.650692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:52.651162 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:52.651721 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:52.651738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:52.652982 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:52.652986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:52.654229 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:52.654242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:52.655093 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:52.655097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:52.655303 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:52.655974 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:52.656005 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:52.658546 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:52.661358 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:52.661366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:52.661382 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:52.661888 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:52.662216 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:52.662220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:52.662223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:52.662440 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:52.662985 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:52.662992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:52.663375 (+ 383us) client_negotiation.cc:769] Sending connection context
0926 19:43:52.663649 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 19:43:52.664116 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":302,"threads_started":1}
I20260926 19:43:52.666821 21231 heartbeater.cc:347] Connected to a master server at 127.2.164.126:46623
I20260926 19:43:52.667115 21231 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:52.667788 21231 heartbeater.cc:511] Master 127.2.164.126:46623 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260926 19:43:52.669216 20908 ts_manager.cc:194] Registered new tserver with Master: 3deb5533ae344dbcab212dcb97460a85 (127.2.164.66:37657)
I20260926 19:43:52.670025 20908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:42469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:43:52.697943 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46623
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:52.998302 21236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:52.998557 21236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:52.998659 21236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:53.005637 21236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:53.005738 21236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:53.005860 21236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:43:53.015187 21236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:46623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:53.016326 21236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:53.017520 21236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:53.029443 21242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:53.029659 21241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:53.029556 21244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:53.030535 21236 server_base.cc:1034] running on GCE node
I20260926 19:43:53.031088 21236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:53.031951 21236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:53.033188 21236 hybrid_clock.cc:648] HybridClock initialized: now 1790451833033118 us; error 30 us; skew 500 ppm
I20260926 19:43:53.033516 21236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:53 dist-test-slave-mrw4 krb5kdc[20863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451833, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:53.036458 21236 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:43:53.037832 21236 webserver.cc:460] Webserver started at http://127.2.164.67:38507/ using document root <none> and password file <none>
I20260926 19:43:53.038429 21236 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:53.038527 21236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:53.038785 21236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:53.041513 21236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "e04f2e36a59f4ac19225c76ba50b2c2a"
format_stamp: "Formatted at 2026-09-26 19:43:53 on dist-test-slave-mrw4"
server_key: "ad60280c7da4b5f75f31469a8a54a680"
server_key_iv: "4824a56ea76bde846d7a7d94ee69c97e"
server_key_version: "encryptionkey@0"
I20260926 19:43:53.042279 21236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "e04f2e36a59f4ac19225c76ba50b2c2a"
format_stamp: "Formatted at 2026-09-26 19:43:53 on dist-test-slave-mrw4"
server_key: "ad60280c7da4b5f75f31469a8a54a680"
server_key_iv: "4824a56ea76bde846d7a7d94ee69c97e"
server_key_version: "encryptionkey@0"
I20260926 19:43:53.047171 21236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:43:53.050436 21251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:53.051647 21236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:43:53.051791 21236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "e04f2e36a59f4ac19225c76ba50b2c2a"
format_stamp: "Formatted at 2026-09-26 19:43:53 on dist-test-slave-mrw4"
server_key: "ad60280c7da4b5f75f31469a8a54a680"
server_key_iv: "4824a56ea76bde846d7a7d94ee69c97e"
server_key_version: "encryptionkey@0"
I20260926 19:43:53.051959 21236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:53.089414 21236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:53.093699 21236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:53.093954 21236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:53.094897 21236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:53.096200 21236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:53.096283 21236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:53.096396 21236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:53.096448 21236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:53.124892 21236 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:33907
I20260926 19:43:53.124953 21363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:33907 every 8 connection(s)
I20260926 19:43:53.126108 21236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:43:53.128017 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 21236
I20260926 19:43:53.128157 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:43:53.128684 2705 external_mini_cluster.cc:1428] Setting key 874a0226578e9fdd751b6cb0a07e8caa
Sep 26 19:43:53 dist-test-slave-mrw4 krb5kdc[20863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451833, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:53.144776 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.128600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:59761 (local address 127.2.164.126:46623)
0926 19:43:53.128805 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:53.128812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:53.130123 (+ 1311us) server_negotiation.cc:407] Connection header received
0926 19:43:53.131913 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:53.131917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:53.131981 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:53.132055 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:53.133462 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.134170 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.135695 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.136089 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.136174 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.139396 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:53.139402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:53.139406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:53.139440 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:53.141878 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:53.142700 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:53.142703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:53.142706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:53.142814 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:53.143193 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:53.143196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:53.143198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:53.143505 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:53.143638 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:53.144251 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 19:43:53.144460 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:43:53.145642 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.129133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:46623 (local address 127.2.164.67:59761)
0926 19:43:53.129976 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:53.130025 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:53.131661 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:53.132246 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:53.132254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:53.132747 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:53.133321 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.133340 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.134340 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.134344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.135580 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.135593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.136264 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.136268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.136485 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.137111 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:53.137133 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:53.139202 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:53.142003 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:53.142012 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:53.142027 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:53.142589 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:53.142908 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:53.142913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:53.142917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:53.143084 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:53.143623 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:53.143631 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:53.143917 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 19:43:53.144141 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 19:43:53.144607 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":322,"threads_started":1}
I20260926 19:43:53.146901 21364 heartbeater.cc:347] Connected to a master server at 127.2.164.126:46623
I20260926 19:43:53.147178 21364 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:53.147794 21364 heartbeater.cc:511] Master 127.2.164.126:46623 requested a full tablet report, sending...
I20260926 19:43:53.149084 20908 ts_manager.cc:194] Registered new tserver with Master: e04f2e36a59f4ac19225c76ba50b2c2a (127.2.164.67:33907)
I20260926 19:43:53.149876 20908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:59761
I20260926 19:43:53.155233 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:43:53.170843 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.158914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.2.164.126:46623)
0926 19:43:53.159071 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:53.159077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:53.159458 (+ 381us) server_negotiation.cc:407] Connection header received
0926 19:43:53.159615 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:53.159621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:53.159681 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:53.159755 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:53.161015 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.161786 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.163053 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.163441 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.163542 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.166023 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:53.166028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:53.166031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:53.166059 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:53.168295 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:53.168985 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:53.168991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:53.168995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:53.169109 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:53.169542 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:53.169546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:53.169548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:53.169816 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:53.170062 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:53.170422 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 19:43:53.170582 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 19:43:53.174955 20908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:43:53.182230 20908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:43:53.203317 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.192835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:38607 (local address 127.0.0.1:42938)
0926 19:43:53.193921 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:53.193953 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:53.194108 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:53.195095 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:53.195104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:53.195136 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:53.195586 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.195599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.197636 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.197642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.200071 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.200087 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.202331 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.202338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.202552 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.202602 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 19:43:53.202732 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 19:43:53.202920 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":758,"threads_started":1}
I20260926 19:43:53.203979 21097 heartbeater.cc:503] Master 127.2.164.126:46623 was elected leader, sending a full tablet report...
I20260926 19:43:53.204293 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.192947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.2.164.65:38607)
0926 19:43:53.193568 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:53.193577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:53.194283 (+ 706us) server_negotiation.cc:407] Connection header received
0926 19:43:53.194379 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:53.194387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:53.194659 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:53.194839 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:53.195775 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.197470 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.200272 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.201828 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.201961 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.202198 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:53.203821 (+ 1623us) server_negotiation.cc:299] Negotiation successful
0926 19:43:53.203977 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":321,"threads_started":1}
I20260926 19:43:53.209481 21032 tablet_service.cc:1467] Processing CreateTablet for tablet b476da8cd9544b86a464e3d215789100 (DEFAULT_TABLE table=test-table [id=62f295e6b10a419c899e0971050d2137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:53.211000 21032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b476da8cd9544b86a464e3d215789100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:53.211369 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.193085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:37657 (local address 127.0.0.1:47526)
0926 19:43:53.203541 (+ 10456us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:53.203556 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:53.203662 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:53.204355 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:53.204360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:53.204388 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:53.204588 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.204600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.206873 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.206879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.208672 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.208687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.210676 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.210682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.210852 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.210882 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 19:43:53.210988 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 19:43:53.211118 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10364,"thread_start_us":14984,"threads_started":1}
I20260926 19:43:53.211534 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.193649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.2.164.66:37657)
0926 19:43:53.194173 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:53.194182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:53.203640 (+ 9458us) server_negotiation.cc:407] Connection header received
0926 19:43:53.203808 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:53.203818 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:53.204095 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:53.204791 (+ 696us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:53.205025 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.206679 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.208835 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.210273 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.210402 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.210583 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:53.211058 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 19:43:53.211224 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":630,"threads_started":1}
I20260926 19:43:53.214453 21166 tablet_service.cc:1467] Processing CreateTablet for tablet b476da8cd9544b86a464e3d215789100 (DEFAULT_TABLE table=test-table [id=62f295e6b10a419c899e0971050d2137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:53.215852 21166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b476da8cd9544b86a464e3d215789100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:53.216550 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.194625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.2.164.67:33907)
0926 19:43:53.208761 (+ 14136us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:53.208770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:53.208802 (+ 32us) server_negotiation.cc:407] Connection header received
0926 19:43:53.208880 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:53.208885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:53.209051 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:53.209231 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:53.209765 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.211731 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.213922 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.215250 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.215380 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.215595 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:53.216037 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 19:43:53.216203 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13970,"thread_start_us":13820,"threads_started":1}
I20260926 19:43:53.217020 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.194309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:33907 (local address 127.0.0.1:48180)
0926 19:43:53.204817 (+ 10508us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:53.204833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:53.204931 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:53.209260 (+ 4329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:53.209264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:53.209288 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:53.209602 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.209610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.211865 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.211870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.213750 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.213765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.215443 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.215450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.215614 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.215647 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:43:53.215727 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 19:43:53.215859 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10399,"thread_start_us":10705,"threads_started":1}
I20260926 19:43:53.221091 21299 tablet_service.cc:1467] Processing CreateTablet for tablet b476da8cd9544b86a464e3d215789100 (DEFAULT_TABLE table=test-table [id=62f295e6b10a419c899e0971050d2137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:53.222477 21299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b476da8cd9544b86a464e3d215789100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:53.224537 21382 tablet_bootstrap.cc:492] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015: Bootstrap starting.
I20260926 19:43:53.228085 21382 tablet_bootstrap.cc:654] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:53.228644 21383 tablet_bootstrap.cc:492] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85: Bootstrap starting.
I20260926 19:43:53.229496 21382 log.cc:824] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:53.231813 21383 tablet_bootstrap.cc:654] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:53.232393 21382 tablet_bootstrap.cc:492] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015: No bootstrap required, opened a new log
I20260926 19:43:53.232664 21382 ts_tablet_manager.cc:1397] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260926 19:43:53.232877 21383 log.cc:824] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:53.235769 21383 tablet_bootstrap.cc:492] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85: No bootstrap required, opened a new log
I20260926 19:43:53.236029 21383 ts_tablet_manager.cc:1397] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260926 19:43:53.236485 21386 tablet_bootstrap.cc:492] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a: Bootstrap starting.
I20260926 19:43:53.239703 21386 tablet_bootstrap.cc:654] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:53.240798 21386 log.cc:824] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:53.241683 21382 raft_consensus.cc:348] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c15f897b12d4b58af05263d514ef015" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38607 } } peers { permanent_uuid: "3deb5533ae344dbcab212dcb97460a85" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37657 } } peers { permanent_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 33907 } }
I20260926 19:43:53.242007 21382 raft_consensus.cc:374] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:53.242095 21382 raft_consensus.cc:729] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c15f897b12d4b58af05263d514ef015, State: Initialized, Role: FOLLOWER
I20260926 19:43:53.242709 21382 consensus_queue.cc:260] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c15f897b12d4b58af05263d514ef015" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38607 } } peers { permanent_uuid: "3deb5533ae344dbcab212dcb97460a85" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37657 } } peers { permanent_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 33907 } }
I20260926 19:43:53.243237 21383 raft_consensus.cc:348] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c15f897b12d4b58af05263d514ef015" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38607 } } peers { permanent_uuid: "3deb5533ae344dbcab212dcb97460a85" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37657 } } peers { permanent_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 33907 } }
I20260926 19:43:53.243637 21383 raft_consensus.cc:374] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:53.243774 21383 raft_consensus.cc:729] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3deb5533ae344dbcab212dcb97460a85, State: Initialized, Role: FOLLOWER
I20260926 19:43:53.244549 21386 tablet_bootstrap.cc:492] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a: No bootstrap required, opened a new log
I20260926 19:43:53.244375 21383 consensus_queue.cc:260] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c15f897b12d4b58af05263d514ef015" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38607 } } peers { permanent_uuid: "3deb5533ae344dbcab212dcb97460a85" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37657 } } peers { permanent_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 33907 } }
I20260926 19:43:53.244604 21382 ts_tablet_manager.cc:1428] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260926 19:43:53.244788 21386 ts_tablet_manager.cc:1397] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 19:43:53.245451 21231 heartbeater.cc:503] Master 127.2.164.126:46623 was elected leader, sending a full tablet report...
I20260926 19:43:53.246019 21383 ts_tablet_manager.cc:1428] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260926 19:43:53.252821 21386 raft_consensus.cc:348] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c15f897b12d4b58af05263d514ef015" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38607 } } peers { permanent_uuid: "3deb5533ae344dbcab212dcb97460a85" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37657 } } peers { permanent_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 33907 } }
I20260926 19:43:53.253182 21386 raft_consensus.cc:374] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:53.253283 21386 raft_consensus.cc:729] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04f2e36a59f4ac19225c76ba50b2c2a, State: Initialized, Role: FOLLOWER
I20260926 19:43:53.253283 21389 raft_consensus.cc:482] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:43:53.253499 21389 raft_consensus.cc:504] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c15f897b12d4b58af05263d514ef015" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38607 } } peers { permanent_uuid: "3deb5533ae344dbcab212dcb97460a85" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37657 } } peers { permanent_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 33907 } }
I20260926 19:43:53.253856 21386 consensus_queue.cc:260] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c15f897b12d4b58af05263d514ef015" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38607 } } peers { permanent_uuid: "3deb5533ae344dbcab212dcb97460a85" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37657 } } peers { permanent_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 33907 } }
I20260926 19:43:53.254804 21364 heartbeater.cc:503] Master 127.2.164.126:46623 was elected leader, sending a full tablet report...
I20260926 19:43:53.255436 21386 ts_tablet_manager.cc:1428] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260926 19:43:53.257283 21389 leader_election.cc:288] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c15f897b12d4b58af05263d514ef015 (127.2.164.65:38607), e04f2e36a59f4ac19225c76ba50b2c2a (127.2.164.67:33907)
I20260926 19:43:53.263026 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.255764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:50557 (local address 127.2.164.65:38607)
0926 19:43:53.255925 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:53.255931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:53.257937 (+ 2006us) server_negotiation.cc:407] Connection header received
0926 19:43:53.258170 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:53.258175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:53.258247 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:53.258307 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:53.259061 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.259758 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.261458 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.262207 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.262342 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.262424 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:53.262685 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 19:43:53.262804 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 19:43:53.263185 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.255408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:38607 (local address 127.2.164.66:50557)
0926 19:43:53.257854 (+ 2446us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:53.257873 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:53.258001 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:53.258517 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:53.258522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:53.258548 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:53.258920 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.258933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.259898 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.259903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.261310 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.261322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.262396 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.262402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.262550 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.262573 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:43:53.262651 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 19:43:53.262768 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2317,"mutex_wait_us":3,"spinlock_wait_cycles":15744,"thread_start_us":2186,"threads_started":1}
I20260926 19:43:53.263707 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.258358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:42513 (local address 127.2.164.67:33907)
0926 19:43:53.258523 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:53.258528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:53.258592 (+ 64us) server_negotiation.cc:407] Connection header received
0926 19:43:53.258978 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:53.258982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:53.259050 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:53.259125 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:53.259610 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.260437 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.262076 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.262801 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.262946 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.263012 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:53.263379 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 19:43:53.263500 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:43:53.263708 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.257982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:33907 (local address 127.2.164.66:42513)
0926 19:43:53.258500 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:53.258517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:53.258710 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:53.259149 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:53.259153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:53.259175 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:53.259473 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.259481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.260559 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.260565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.261940 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:53.261952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.262993 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.262998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:53.263198 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.263227 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 19:43:53.263349 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 19:43:53.263476 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":542,"threads_started":1}
I20260926 19:43:53.264075 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476da8cd9544b86a464e3d215789100" candidate_uuid: "3deb5533ae344dbcab212dcb97460a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c15f897b12d4b58af05263d514ef015" is_pre_election: true
I20260926 19:43:53.264487 21052 raft_consensus.cc:2454] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3deb5533ae344dbcab212dcb97460a85 in term 0.
I20260926 19:43:53.265035 21122 leader_election.cc:302] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c15f897b12d4b58af05263d514ef015, 3deb5533ae344dbcab212dcb97460a85; no voters:
I20260926 19:43:53.265389 21389 raft_consensus.cc:2781] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:43:53.265516 21389 raft_consensus.cc:482] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:43:53.265606 21389 raft_consensus.cc:3037] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:53.266055 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476da8cd9544b86a464e3d215789100" candidate_uuid: "3deb5533ae344dbcab212dcb97460a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" is_pre_election: true
I20260926 19:43:53.266510 21319 raft_consensus.cc:2454] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3deb5533ae344dbcab212dcb97460a85 in term 0.
I20260926 19:43:53.268105 21389 raft_consensus.cc:504] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c15f897b12d4b58af05263d514ef015" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38607 } } peers { permanent_uuid: "3deb5533ae344dbcab212dcb97460a85" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37657 } } peers { permanent_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 33907 } }
I20260926 19:43:53.268651 21389 leader_election.cc:288] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [CANDIDATE]: Term 1 election: Requested vote from peers 0c15f897b12d4b58af05263d514ef015 (127.2.164.65:38607), e04f2e36a59f4ac19225c76ba50b2c2a (127.2.164.67:33907)
I20260926 19:43:53.269001 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476da8cd9544b86a464e3d215789100" candidate_uuid: "3deb5533ae344dbcab212dcb97460a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c15f897b12d4b58af05263d514ef015"
I20260926 19:43:53.269054 21319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476da8cd9544b86a464e3d215789100" candidate_uuid: "3deb5533ae344dbcab212dcb97460a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a"
I20260926 19:43:53.269157 21052 raft_consensus.cc:3037] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:53.269243 21319 raft_consensus.cc:3037] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:53.271400 21052 raft_consensus.cc:2454] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3deb5533ae344dbcab212dcb97460a85 in term 1.
I20260926 19:43:53.271554 21319 raft_consensus.cc:2454] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3deb5533ae344dbcab212dcb97460a85 in term 1.
I20260926 19:43:53.271750 21122 leader_election.cc:302] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c15f897b12d4b58af05263d514ef015, 3deb5533ae344dbcab212dcb97460a85; no voters:
I20260926 19:43:53.272214 21389 raft_consensus.cc:2781] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:43:53.272572 21389 raft_consensus.cc:686] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 1 LEADER]: Becoming Leader. State: Replica: 3deb5533ae344dbcab212dcb97460a85, State: Running, Role: LEADER
I20260926 19:43:53.272979 21389 consensus_queue.cc:237] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c15f897b12d4b58af05263d514ef015" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38607 } } peers { permanent_uuid: "3deb5533ae344dbcab212dcb97460a85" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37657 } } peers { permanent_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 33907 } }
I20260926 19:43:53.277287 20907 catalog_manager.cc:5495] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3deb5533ae344dbcab212dcb97460a85 (127.2.164.66). New cstate: current_term: 1 leader_uuid: "3deb5533ae344dbcab212dcb97460a85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c15f897b12d4b58af05263d514ef015" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3deb5533ae344dbcab212dcb97460a85" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 33907 } health_report { overall_health: UNKNOWN } } }
I20260926 19:43:53.302400 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.297352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.2.164.66:37657)
0926 19:43:53.297556 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:53.297562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:53.297640 (+ 78us) server_negotiation.cc:407] Connection header received
0926 19:43:53.297796 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:53.297800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:53.297886 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:53.297999 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:43:53.298295 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.298973 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.300312 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.300710 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.300809 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.301387 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:43:53.301854 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:43:53.301979 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:53.302090 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 19:43:53.302129 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:43:53.310278 21319 raft_consensus.cc:1261] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a [term 1 FOLLOWER]: Refusing update from remote peer 3deb5533ae344dbcab212dcb97460a85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:53.310278 21052 raft_consensus.cc:1261] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015 [term 1 FOLLOWER]: Refusing update from remote peer 3deb5533ae344dbcab212dcb97460a85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:53.311131 21393 consensus_queue.cc:1035] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c15f897b12d4b58af05263d514ef015" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 38607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:43:53.311427 21389 consensus_queue.cc:1035] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04f2e36a59f4ac19225c76ba50b2c2a" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 33907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:43:53.335142 20908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:53.335386 20908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:53.338583 20908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fadb9e37a84847cf8669c024a20773bd: Sending DeleteTablet for 3 replicas of tablet b476da8cd9544b86a464e3d215789100
I20260926 19:43:53.339570 21166 tablet_service.cc:1514] Processing DeleteTablet for tablet b476da8cd9544b86a464e3d215789100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:53 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:47526
I20260926 19:43:53.339723 21299 tablet_service.cc:1514] Processing DeleteTablet for tablet b476da8cd9544b86a464e3d215789100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:53 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:48180
I20260926 19:43:53.339915 21032 tablet_service.cc:1514] Processing DeleteTablet for tablet b476da8cd9544b86a464e3d215789100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:53 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:42938
I20260926 19:43:53.340337 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 20969
I20260926 19:43:53.343230 21407 tablet_replica.cc:330] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a: stopping tablet replica
I20260926 19:43:53.344106 21408 tablet_replica.cc:330] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015: stopping tablet replica
I20260926 19:43:53.344383 21407 raft_consensus.cc:2229] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:53.344491 21408 raft_consensus.cc:2229] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:53.344933 21408 raft_consensus.cc:2258] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:53.344933 21407 raft_consensus.cc:2258] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:53.345727 21406 tablet_replica.cc:330] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85: stopping tablet replica
I20260926 19:43:53.346105 21406 raft_consensus.cc:2229] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:43:53.346781 21406 raft_consensus.cc:2258] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:53.347158 21407 ts_tablet_manager.cc:1905] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:53.347424 21408 ts_tablet_manager.cc:1905] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:53.353874 21407 ts_tablet_manager.cc:1918] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:53.354022 21407 log.cc:1196] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/b476da8cd9544b86a464e3d215789100
I20260926 19:43:53.354112 21406 ts_tablet_manager.cc:1905] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:53.354507 21407 ts_tablet_manager.cc:1939] T b476da8cd9544b86a464e3d215789100 P e04f2e36a59f4ac19225c76ba50b2c2a: Deleting consensus metadata
Sep 26 19:43:53 dist-test-slave-mrw4 krb5kdc[20863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:43:53.356038 20895 catalog_manager.cc:4841] TS e04f2e36a59f4ac19225c76ba50b2c2a (127.2.164.67:33907): tablet b476da8cd9544b86a464e3d215789100 (table test-table [id=62f295e6b10a419c899e0971050d2137]) successfully deleted
I20260926 19:43:53.356899 21408 ts_tablet_manager.cc:1918] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:53.357055 21408 log.cc:1196] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/b476da8cd9544b86a464e3d215789100
I20260926 19:43:53.357653 21408 ts_tablet_manager.cc:1939] T b476da8cd9544b86a464e3d215789100 P 0c15f897b12d4b58af05263d514ef015: Deleting consensus metadata
I20260926 19:43:53.359304 20895 catalog_manager.cc:4841] TS 0c15f897b12d4b58af05263d514ef015 (127.2.164.65:38607): tablet b476da8cd9544b86a464e3d215789100 (table test-table [id=62f295e6b10a419c899e0971050d2137]) successfully deleted
I20260926 19:43:53.362216 21406 ts_tablet_manager.cc:1918] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:53.362383 21406 log.cc:1196] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/b476da8cd9544b86a464e3d215789100
I20260926 19:43:53.362870 21406 ts_tablet_manager.cc:1939] T b476da8cd9544b86a464e3d215789100 P 3deb5533ae344dbcab212dcb97460a85: Deleting consensus metadata
I20260926 19:43:53.363139 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.341466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.2.164.65:38607)
0926 19:43:53.341635 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:53.341641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:53.345592 (+ 3951us) server_negotiation.cc:407] Connection header received
0926 19:43:53.345845 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:53.345851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:53.345931 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:53.346017 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:53.347860 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.348968 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.351169 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.351783 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.351881 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.357119 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:53.357132 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:53.357149 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:53.357208 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:53.360390 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:53.361036 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:53.361042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:53.361044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:53.361191 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:53.361565 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:53.361569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:53.361571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:53.361906 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:53.362159 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:53.362595 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 19:43:53.362785 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:43:53.363720 21092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:53.364378 20894 catalog_manager.cc:4841] TS 3deb5533ae344dbcab212dcb97460a85 (127.2.164.66:37657): tablet b476da8cd9544b86a464e3d215789100 (table test-table [id=62f295e6b10a419c899e0971050d2137]) successfully deleted
I20260926 19:43:53.460224 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 20969
I20260926 19:43:53.475191 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 21103
Sep 26 19:43:53 dist-test-slave-mrw4 krb5kdc[20863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:43:53.497594 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.475922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.2.164.66:37657)
0926 19:43:53.476108 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:53.476113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:53.476138 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:43:53.476269 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:53.476275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:53.476358 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:53.476458 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:53.477794 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.478984 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.480261 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.480887 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.481013 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.490917 (+ 9904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:53.490936 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:53.490956 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:53.490999 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:53.494830 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:53.495502 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:53.495512 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:53.495516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:53.495685 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:53.496001 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:53.496006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:53.496010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:53.496482 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:53.496809 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:53.496956 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 19:43:53.497200 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:43:53.498353 21226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:53.611601 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 21103
I20260926 19:43:53.631870 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 21236
Sep 26 19:43:53 dist-test-slave-mrw4 krb5kdc[20863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:43:53.646593 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:53.632643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.2.164.67:33907)
0926 19:43:53.632812 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:53.632817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:53.632851 (+ 34us) server_negotiation.cc:407] Connection header received
0926 19:43:53.633066 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:53.633070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:53.633137 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:53.633270 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:53.634562 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.635291 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.636570 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:53.636967 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:53.637061 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:53.641284 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:53.641298 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:53.641314 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:53.641348 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:53.644063 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:53.644695 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:53.644702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:53.644704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:53.644826 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:53.645103 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:53.645106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:53.645108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:53.645582 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:53.645792 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:53.646123 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 19:43:53.646296 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:43:53.647238 21359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:53.735364 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 21236
I20260926 19:43:53.748880 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 20876
I20260926 19:43:53.749953 20938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:53.856819 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 20876
2026-09-26T19:43:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3882 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T19:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:43:53Z Disabled control of system clock
I20260926 19:43:53.922387 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:32883
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:45161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:32883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:43:54.231864 21417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:54.232143 21417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:54.232268 21417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:54.239292 21417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:43:54.239408 21417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:54.239470 21417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:54.239545 21417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:43:54.239602 21417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:43:54.248467 21417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:32883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:54.249760 21417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:54.251451 21417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:54.262341 21422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:54.262269 21425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:54.262306 21423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:54.263309 21417 server_base.cc:1034] running on GCE node
I20260926 19:43:54.264045 21417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:54.265374 21417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:54.266582 21417 hybrid_clock.cc:648] HybridClock initialized: now 1790451834266538 us; error 41 us; skew 500 ppm
I20260926 19:43:54.266913 21417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:43:54.268976 21417 webserver.cc:460] Webserver started at http://127.2.164.126:41497/ using document root <none> and password file <none>
I20260926 19:43:54.269661 21417 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:54.269774 21417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:54.270063 21417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:54.272784 21417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "9acb249beac243e6933bc11175705ec0"
format_stamp: "Formatted at 2026-09-26 19:43:54 on dist-test-slave-mrw4"
server_key: "d2b9914f795db5168cb684a4a636f25f"
server_key_iv: "5e50a9d0cba85626fba27ed530ecdb88"
server_key_version: "encryptionkey@0"
I20260926 19:43:54.273559 21417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "9acb249beac243e6933bc11175705ec0"
format_stamp: "Formatted at 2026-09-26 19:43:54 on dist-test-slave-mrw4"
server_key: "d2b9914f795db5168cb684a4a636f25f"
server_key_iv: "5e50a9d0cba85626fba27ed530ecdb88"
server_key_version: "encryptionkey@0"
I20260926 19:43:54.278285 21417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:43:54.281481 21431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:54.282668 21417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 19:43:54.282832 21417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "9acb249beac243e6933bc11175705ec0"
format_stamp: "Formatted at 2026-09-26 19:43:54 on dist-test-slave-mrw4"
server_key: "d2b9914f795db5168cb684a4a636f25f"
server_key_iv: "5e50a9d0cba85626fba27ed530ecdb88"
server_key_version: "encryptionkey@0"
I20260926 19:43:54.283015 21417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:54.299441 21417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:54.300220 21417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:54.300467 21417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:54.316601 21417 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:32883
I20260926 19:43:54.316620 21482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:32883 every 8 connection(s)
I20260926 19:43:54.317829 21417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:43:54.320555 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 21417
I20260926 19:43:54.320731 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:43:54.321287 2705 external_mini_cluster.cc:1428] Setting key f893bb6553779f3ca69cae8e8c1cd875
I20260926 19:43:54.321583 21483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:54.333526 21483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0: Bootstrap starting.
I20260926 19:43:54.334517 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:54.323169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.2.164.126:32883)
0926 19:43:54.324089 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:54.324103 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:54.324144 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:43:54.326039 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:54.326059 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:54.326495 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:54.326847 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:54.327387 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:54.328665 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:54.330023 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:54.330714 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:54.330894 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:54.332124 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:54.332131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:54.332144 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:54.332777 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:54.332851 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:54.332872 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:54.333073 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:54.333293 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 19:43:54.333588 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":521,"thread_start_us":326,"threads_started":1}
I20260926 19:43:54.336861 21483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:54.337980 21483 log.cc:824] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:54.340372 21483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0: No bootstrap required, opened a new log
I20260926 19:43:54.347158 21483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acb249beac243e6933bc11175705ec0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 32883 } }
I20260926 19:43:54.347472 21483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:54.347560 21483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9acb249beac243e6933bc11175705ec0, State: Initialized, Role: FOLLOWER
I20260926 19:43:54.348060 21483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acb249beac243e6933bc11175705ec0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 32883 } }
I20260926 19:43:54.348337 21483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:43:54.348480 21483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:43:54.348632 21483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:54.350652 21483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acb249beac243e6933bc11175705ec0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 32883 } }
I20260926 19:43:54.351089 21483 leader_election.cc:302] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9acb249beac243e6933bc11175705ec0; no voters:
I20260926 19:43:54.351585 21483 leader_election.cc:288] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:43:54.351706 21488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:43:54.352398 21488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [term 1 LEADER]: Becoming Leader. State: Replica: 9acb249beac243e6933bc11175705ec0, State: Running, Role: LEADER
I20260926 19:43:54.352865 21488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acb249beac243e6933bc11175705ec0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 32883 } }
I20260926 19:43:54.353544 21483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:43:54.355481 21489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9acb249beac243e6933bc11175705ec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acb249beac243e6933bc11175705ec0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 32883 } } }
I20260926 19:43:54.355504 21490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9acb249beac243e6933bc11175705ec0. Latest consensus state: current_term: 1 leader_uuid: "9acb249beac243e6933bc11175705ec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acb249beac243e6933bc11175705ec0" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 32883 } } }
I20260926 19:43:54.355931 21490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:54.357551 21489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0 [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:54.357825 21494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:43:54.363654 21494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:43:54.372217 21494 catalog_manager.cc:1347] Generated new cluster ID: 0d60999c1cc4409c9a273a42cb408000
I20260926 19:43:54.372310 21494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:43:54.397468 21494 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:43:54.398903 21494 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:43:54.407616 21494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0: Generated new TSK 0
I20260926 19:43:54.408402 21494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:43:54.414065 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:32883
--builtin_ntp_servers=127.2.164.84:45161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:43:54.725066 21507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:54.725350 21507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:54.725486 21507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:54.732436 21507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:54.732529 21507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:54.732661 21507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:43:54.743625 21507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:54.745044 21507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:54.746754 21507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:54.758486 21512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:54.758535 21513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:54.758558 21515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:54.759649 21507 server_base.cc:1034] running on GCE node
I20260926 19:43:54.760208 21507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:54.761071 21507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:54.762287 21507 hybrid_clock.cc:648] HybridClock initialized: now 1790451834762238 us; error 40 us; skew 500 ppm
I20260926 19:43:54.762585 21507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:43:54.764616 21507 webserver.cc:460] Webserver started at http://127.2.164.65:42065/ using document root <none> and password file <none>
I20260926 19:43:54.765273 21507 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:54.765372 21507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:54.765651 21507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:54.768258 21507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8"
format_stamp: "Formatted at 2026-09-26 19:43:54 on dist-test-slave-mrw4"
server_key: "85e2d7dc3c7fad56125ccc9cd99010e8"
server_key_iv: "a489590f70b5627d4c93e76f57e99e43"
server_key_version: "encryptionkey@0"
I20260926 19:43:54.768980 21507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8"
format_stamp: "Formatted at 2026-09-26 19:43:54 on dist-test-slave-mrw4"
server_key: "85e2d7dc3c7fad56125ccc9cd99010e8"
server_key_iv: "a489590f70b5627d4c93e76f57e99e43"
server_key_version: "encryptionkey@0"
I20260926 19:43:54.773689 21507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:43:54.776875 21521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:54.778148 21507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:43:54.778290 21507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8"
format_stamp: "Formatted at 2026-09-26 19:43:54 on dist-test-slave-mrw4"
server_key: "85e2d7dc3c7fad56125ccc9cd99010e8"
server_key_iv: "a489590f70b5627d4c93e76f57e99e43"
server_key_version: "encryptionkey@0"
I20260926 19:43:54.778481 21507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:54.815490 21507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:54.816238 21507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:54.816516 21507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:54.817411 21507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:54.818742 21507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:54.818833 21507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:54.818930 21507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:54.818995 21507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:54.847384 21507 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:43577
I20260926 19:43:54.847433 21633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:43577 every 8 connection(s)
I20260926 19:43:54.848592 21507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:43:54.854941 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 21507
I20260926 19:43:54.855088 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:43:54.855532 2705 external_mini_cluster.cc:1428] Setting key afc8fdf61655877c3876e6b6f3ba3ac2
I20260926 19:43:54.858491 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:32883
--builtin_ntp_servers=127.2.164.84:45161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:43:54.861485 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:54.851025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:36115 (local address 127.2.164.126:32883)
0926 19:43:54.851440 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:54.851444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:54.852564 (+ 1120us) server_negotiation.cc:407] Connection header received
0926 19:43:54.854428 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:54.854435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:54.854539 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:54.854644 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:54.855754 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:54.856638 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:54.858820 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:54.859318 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:54.859422 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:54.860789 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:54.860794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:54.860797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:54.860813 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:54.860858 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:54.860871 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:54.860940 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:54.861138 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 19:43:54.861240 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":217,"threads_started":1}
I20260926 19:43:54.862686 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:54.851532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:32883 (local address 127.2.164.65:36115)
0926 19:43:54.852409 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:54.852449 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:54.854152 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:54.854884 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:54.854900 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:54.854997 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:54.855600 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:54.855619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:54.856814 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:54.856819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:54.858660 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:54.858676 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:54.859527 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:54.859535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:54.859825 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:54.860465 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:43:54.860468 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:54.860516 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:43:54.860612 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:54.860954 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:54.860962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:54.860969 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 19:43:54.861131 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 19:43:54.861515 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":284,"threads_started":1}
I20260926 19:43:54.864219 21634 heartbeater.cc:347] Connected to a master server at 127.2.164.126:32883
I20260926 19:43:54.864502 21634 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:54.865249 21634 heartbeater.cc:511] Master 127.2.164.126:32883 requested a full tablet report, sending...
I20260926 19:43:54.867224 21448 ts_manager.cc:194] Registered new tserver with Master: c79d38c757e54ab5bd6bd592dc5f1ff8 (127.2.164.65:43577)
I20260926 19:43:54.868649 21448 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.65:36115
W20260926 19:43:55.164705 21638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:55.164959 21638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:55.165066 21638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:55.171972 21638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:55.172060 21638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:55.172190 21638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:43:55.180788 21638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:55.182037 21638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:55.183749 21638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:55.195901 21646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:55.195973 21643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:55.195956 21644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:55.196898 21638 server_base.cc:1034] running on GCE node
I20260926 19:43:55.197479 21638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:55.198328 21638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:55.199529 21638 hybrid_clock.cc:648] HybridClock initialized: now 1790451835199487 us; error 32 us; skew 500 ppm
I20260926 19:43:55.199810 21638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:43:55.202067 21638 webserver.cc:460] Webserver started at http://127.2.164.66:38965/ using document root <none> and password file <none>
I20260926 19:43:55.202713 21638 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:55.202813 21638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:55.203105 21638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:55.205878 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "06b6ba84d3d9445ea4d1d48e12d74373"
format_stamp: "Formatted at 2026-09-26 19:43:55 on dist-test-slave-mrw4"
server_key: "86676656ab42e7044e2709c4450cbb29"
server_key_iv: "ffa9044dfc3850b401b1920c9afb5cdc"
server_key_version: "encryptionkey@0"
I20260926 19:43:55.206610 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "06b6ba84d3d9445ea4d1d48e12d74373"
format_stamp: "Formatted at 2026-09-26 19:43:55 on dist-test-slave-mrw4"
server_key: "86676656ab42e7044e2709c4450cbb29"
server_key_iv: "ffa9044dfc3850b401b1920c9afb5cdc"
server_key_version: "encryptionkey@0"
I20260926 19:43:55.211427 21638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:43:55.214814 21652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:55.216094 21638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:43:55.216246 21638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "06b6ba84d3d9445ea4d1d48e12d74373"
format_stamp: "Formatted at 2026-09-26 19:43:55 on dist-test-slave-mrw4"
server_key: "86676656ab42e7044e2709c4450cbb29"
server_key_iv: "ffa9044dfc3850b401b1920c9afb5cdc"
server_key_version: "encryptionkey@0"
I20260926 19:43:55.216447 21638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:55.252204 21638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:55.253000 21638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:55.253325 21638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:55.254240 21638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:55.255640 21638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:55.255730 21638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:55.255832 21638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:55.255899 21638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:55.284086 21638 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:37703
I20260926 19:43:55.284132 21764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:37703 every 8 connection(s)
I20260926 19:43:55.285351 21638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:43:55.292094 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 21638
I20260926 19:43:55.292248 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:43:55.292698 2705 external_mini_cluster.cc:1428] Setting key ac4d4c7c8168cd2e640d23ee6f269103
I20260926 19:43:55.295269 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:32883
--builtin_ntp_servers=127.2.164.84:45161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:43:55.297776 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.287935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:53355 (local address 127.2.164.126:32883)
0926 19:43:55.288106 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:55.288111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:55.289625 (+ 1514us) server_negotiation.cc:407] Connection header received
0926 19:43:55.291421 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:55.291427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:55.291510 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:55.291596 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:55.292679 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.293538 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.295112 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.295577 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.295687 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.296977 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:55.296983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:55.296986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:55.297002 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:55.297033 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:55.297044 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:55.297115 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:55.297389 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 19:43:55.297484 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:43:55.298725 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.288505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:32883 (local address 127.2.164.66:53355)
0926 19:43:55.289476 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:55.289514 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:55.291147 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:55.291820 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:55.291829 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:55.291914 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:55.292523 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.292541 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.293675 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.293681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.294979 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.294992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.295781 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.295789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.295995 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.296658 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:43:55.296660 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:55.296703 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:43:55.296782 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:55.297128 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:55.297139 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:55.297147 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 19:43:55.297366 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 19:43:55.297733 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":291,"threads_started":1}
I20260926 19:43:55.300019 21765 heartbeater.cc:347] Connected to a master server at 127.2.164.126:32883
I20260926 19:43:55.300318 21765 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:55.301116 21765 heartbeater.cc:511] Master 127.2.164.126:32883 requested a full tablet report, sending...
I20260926 19:43:55.302569 21448 ts_manager.cc:194] Registered new tserver with Master: 06b6ba84d3d9445ea4d1d48e12d74373 (127.2.164.66:37703)
I20260926 19:43:55.303339 21448 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.66:53355
W20260926 19:43:55.604496 21768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:55.604748 21768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:55.604856 21768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:55.611842 21768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:55.611931 21768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:55.612062 21768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:43:55.620779 21768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:55.621966 21768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:55.623625 21768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:55.635205 21773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:55.635273 21774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:55.635298 21776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:55.636341 21768 server_base.cc:1034] running on GCE node
I20260926 19:43:55.636859 21768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:55.637728 21768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:55.638950 21768 hybrid_clock.cc:648] HybridClock initialized: now 1790451835638894 us; error 45 us; skew 500 ppm
I20260926 19:43:55.639252 21768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:43:55.641383 21768 webserver.cc:460] Webserver started at http://127.2.164.67:39619/ using document root <none> and password file <none>
I20260926 19:43:55.642007 21768 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:55.642109 21768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:55.642412 21768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:55.645061 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "cc593831e0704dbb8979dcc4548fadc3"
format_stamp: "Formatted at 2026-09-26 19:43:55 on dist-test-slave-mrw4"
server_key: "de92c5a7516d9bd34bef479e685f7010"
server_key_iv: "1470b95c5194d50acf20e9455d1db3cb"
server_key_version: "encryptionkey@0"
I20260926 19:43:55.646019 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "cc593831e0704dbb8979dcc4548fadc3"
format_stamp: "Formatted at 2026-09-26 19:43:55 on dist-test-slave-mrw4"
server_key: "de92c5a7516d9bd34bef479e685f7010"
server_key_iv: "1470b95c5194d50acf20e9455d1db3cb"
server_key_version: "encryptionkey@0"
I20260926 19:43:55.650677 21768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:43:55.654059 21782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:55.655349 21768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:43:55.655481 21768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "cc593831e0704dbb8979dcc4548fadc3"
format_stamp: "Formatted at 2026-09-26 19:43:55 on dist-test-slave-mrw4"
server_key: "de92c5a7516d9bd34bef479e685f7010"
server_key_iv: "1470b95c5194d50acf20e9455d1db3cb"
server_key_version: "encryptionkey@0"
I20260926 19:43:55.655666 21768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:55.691951 21768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:55.692855 21768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:55.693194 21768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:55.694133 21768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:55.695509 21768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:55.695591 21768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:55.695689 21768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:55.695757 21768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:55.724310 21768 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:46873
I20260926 19:43:55.724339 21894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:46873 every 8 connection(s)
I20260926 19:43:55.725605 21768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:43:55.727205 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 21768
I20260926 19:43:55.727365 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:43:55.727854 2705 external_mini_cluster.cc:1428] Setting key f4b8ef8d7b47b1f961c56db442755a3a
I20260926 19:43:55.737854 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.729268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:56121 (local address 127.2.164.126:32883)
0926 19:43:55.729471 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:55.729477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:55.730046 (+ 569us) server_negotiation.cc:407] Connection header received
0926 19:43:55.731819 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:55.731825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:55.731908 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:55.731993 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:55.733008 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.733780 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.735352 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.735756 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.735864 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.737070 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:55.737075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:55.737078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:55.737095 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:55.737126 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:55.737138 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:55.737253 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:55.737445 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 19:43:55.737538 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:43:55.738729 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.728827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:32883 (local address 127.2.164.67:56121)
0926 19:43:55.729904 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:55.729953 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:55.731571 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:55.732217 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:55.732225 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:55.732308 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:55.732836 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.732853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.733938 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.733942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.735218 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.735231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.735965 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.735974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.736174 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.736767 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:43:55.736770 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:55.736804 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:43:55.736879 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:55.737254 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:55.737265 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:55.737273 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 19:43:55.737418 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 19:43:55.737774 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":371,"threads_started":1}
I20260926 19:43:55.739900 21895 heartbeater.cc:347] Connected to a master server at 127.2.164.126:32883
I20260926 19:43:55.740135 21895 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:55.740754 21895 heartbeater.cc:511] Master 127.2.164.126:32883 requested a full tablet report, sending...
I20260926 19:43:55.742033 21448 ts_manager.cc:194] Registered new tserver with Master: cc593831e0704dbb8979dcc4548fadc3 (127.2.164.67:46873)
I20260926 19:43:55.742879 21448 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.67:56121
I20260926 19:43:55.743005 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:43:55.752251 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.746843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.2.164.126:32883)
0926 19:43:55.747069 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:55.747075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:55.747346 (+ 271us) server_negotiation.cc:407] Connection header received
0926 19:43:55.747467 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:55.747471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:55.747535 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:55.747607 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:55.748053 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.748769 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.750132 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.750531 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.750627 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.751616 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:55.751621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:55.751624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:55.751641 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:55.751671 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:55.751682 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:55.751787 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:55.751910 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 19:43:55.751950 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:43:55.755786 21448 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:43:55.762369 21448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:43:55.785071 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.773876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43620 (local address 127.2.164.65:43577)
0926 19:43:55.775081 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:55.775091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:55.775747 (+ 656us) server_negotiation.cc:407] Connection header received
0926 19:43:55.775847 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:55.775854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:55.776094 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:55.776276 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:55.777259 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.778910 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.782334 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.783599 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.783726 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.783984 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:55.784554 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 19:43:55.784723 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":909,"threads_started":1}
I20260926 19:43:55.785095 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.773194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:46873 (local address 127.0.0.1:44712)
0926 19:43:55.774240 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:55.774289 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:55.774476 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:55.776374 (+ 1898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:55.776383 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:55.776415 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:55.776904 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.776918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.780271 (+ 3353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.780277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.782660 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.782674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.784275 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.784281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.784446 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.784477 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:43:55.784547 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 19:43:55.784734 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":4,"thread_start_us":619,"threads_started":1}
I20260926 19:43:55.785133 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.773483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.2.164.67:46873)
0926 19:43:55.775691 (+ 2208us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:55.775699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:55.775735 (+ 36us) server_negotiation.cc:407] Connection header received
0926 19:43:55.775837 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:55.775846 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:55.776112 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:55.777114 (+ 1002us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:55.778529 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.780103 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.782851 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.784083 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.784214 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.784478 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:55.784603 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 19:43:55.784771 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2043,"thread_start_us":1898,"threads_started":1}
I20260926 19:43:55.785989 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.773369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43577 (local address 127.0.0.1:43620)
0926 19:43:55.775493 (+ 2124us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:55.775509 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:55.775630 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:55.776543 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:55.776548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:55.776575 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:55.777028 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.777040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.779080 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.779086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.782141 (+ 3055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.782156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.784101 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.784108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.784281 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.784340 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 19:43:55.785532 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0926 19:43:55.785652 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1984,"mutex_wait_us":560,"thread_start_us":1850,"threads_started":1}
I20260926 19:43:55.788923 21830 tablet_service.cc:1467] Processing CreateTablet for tablet 860b6c9baa12468da58cab8cfc0676f3 (DEFAULT_TABLE table=test-table [id=223ab38186af49ed9946f14a435146f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:55.789438 21569 tablet_service.cc:1467] Processing CreateTablet for tablet 860b6c9baa12468da58cab8cfc0676f3 (DEFAULT_TABLE table=test-table [id=223ab38186af49ed9946f14a435146f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:55.790285 21830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 860b6c9baa12468da58cab8cfc0676f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:55.790534 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.775597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:37703 (local address 127.0.0.1:37122)
0926 19:43:55.782787 (+ 7190us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:55.782805 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:55.782929 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:55.783574 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:55.783579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:55.783607 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:55.783959 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.783971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.786060 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.786175 (+ 115us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.788072 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.788088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.789856 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.789863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.790019 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.790049 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 19:43:55.790136 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 19:43:55.790247 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7066,"thread_start_us":6926,"threads_started":1}
I20260926 19:43:55.791085 21569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 860b6c9baa12468da58cab8cfc0676f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:55.791112 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.776012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.2.164.66:37703)
0926 19:43:55.781337 (+ 5325us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:55.781345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:55.782923 (+ 1578us) server_negotiation.cc:407] Connection header received
0926 19:43:55.783065 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:55.783074 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:55.783372 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:55.783565 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:55.784130 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.785892 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.788280 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.789658 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.789779 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.790019 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:55.790367 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 19:43:55.790539 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5169,"thread_start_us":5033,"threads_started":1}
I20260926 19:43:55.794329 21700 tablet_service.cc:1467] Processing CreateTablet for tablet 860b6c9baa12468da58cab8cfc0676f3 (DEFAULT_TABLE table=test-table [id=223ab38186af49ed9946f14a435146f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:55.795631 21700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 860b6c9baa12468da58cab8cfc0676f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:55.801501 21913 tablet_bootstrap.cc:492] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8: Bootstrap starting.
I20260926 19:43:55.803957 21914 tablet_bootstrap.cc:492] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: Bootstrap starting.
I20260926 19:43:55.805089 21913 tablet_bootstrap.cc:654] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:55.806288 21913 log.cc:824] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:55.807170 21914 tablet_bootstrap.cc:654] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:55.808274 21914 log.cc:824] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:55.809234 21913 tablet_bootstrap.cc:492] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8: No bootstrap required, opened a new log
I20260926 19:43:55.809475 21913 ts_tablet_manager.cc:1397] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260926 19:43:55.811206 21914 tablet_bootstrap.cc:492] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: No bootstrap required, opened a new log
I20260926 19:43:55.811478 21914 ts_tablet_manager.cc:1397] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 19:43:55.811944 21917 tablet_bootstrap.cc:492] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: Bootstrap starting.
I20260926 19:43:55.815110 21917 tablet_bootstrap.cc:654] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:55.816210 21917 log.cc:824] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:55.818487 21913 raft_consensus.cc:348] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc593831e0704dbb8979dcc4548fadc3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46873 } } peers { permanent_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43577 } } peers { permanent_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37703 } }
I20260926 19:43:55.818908 21913 raft_consensus.cc:374] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:55.819010 21917 tablet_bootstrap.cc:492] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: No bootstrap required, opened a new log
I20260926 19:43:55.819008 21913 raft_consensus.cc:729] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79d38c757e54ab5bd6bd592dc5f1ff8, State: Initialized, Role: FOLLOWER
I20260926 19:43:55.819250 21917 ts_tablet_manager.cc:1397] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 19:43:55.819239 21914 raft_consensus.cc:348] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc593831e0704dbb8979dcc4548fadc3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46873 } } peers { permanent_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43577 } } peers { permanent_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37703 } }
I20260926 19:43:55.819626 21914 raft_consensus.cc:374] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:55.819718 21914 raft_consensus.cc:729] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc593831e0704dbb8979dcc4548fadc3, State: Initialized, Role: FOLLOWER
I20260926 19:43:55.819661 21913 consensus_queue.cc:260] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc593831e0704dbb8979dcc4548fadc3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46873 } } peers { permanent_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43577 } } peers { permanent_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37703 } }
I20260926 19:43:55.820226 21914 consensus_queue.cc:260] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc593831e0704dbb8979dcc4548fadc3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46873 } } peers { permanent_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43577 } } peers { permanent_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37703 } }
I20260926 19:43:55.820942 21634 heartbeater.cc:503] Master 127.2.164.126:32883 was elected leader, sending a full tablet report...
I20260926 19:43:55.821559 21913 ts_tablet_manager.cc:1428] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260926 19:43:55.822324 21895 heartbeater.cc:503] Master 127.2.164.126:32883 was elected leader, sending a full tablet report...
I20260926 19:43:55.822896 21914 ts_tablet_manager.cc:1428] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260926 19:43:55.829737 21917 raft_consensus.cc:348] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc593831e0704dbb8979dcc4548fadc3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46873 } } peers { permanent_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43577 } } peers { permanent_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37703 } }
I20260926 19:43:55.830029 21917 raft_consensus.cc:374] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:55.830114 21917 raft_consensus.cc:729] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b6ba84d3d9445ea4d1d48e12d74373, State: Initialized, Role: FOLLOWER
I20260926 19:43:55.830633 21917 consensus_queue.cc:260] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc593831e0704dbb8979dcc4548fadc3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46873 } } peers { permanent_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43577 } } peers { permanent_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37703 } }
I20260926 19:43:55.831462 21765 heartbeater.cc:503] Master 127.2.164.126:32883 was elected leader, sending a full tablet report...
I20260926 19:43:55.831828 21917 ts_tablet_manager.cc:1428] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: Time spent starting tablet: real 0.012s user 0.005s sys 0.004s
W20260926 19:43:55.850528 21635 tablet.cc:2369] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:43:55.937294 21921 raft_consensus.cc:482] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:43:55.937552 21921 raft_consensus.cc:504] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc593831e0704dbb8979dcc4548fadc3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46873 } } peers { permanent_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43577 } } peers { permanent_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37703 } }
I20260926 19:43:55.939121 21921 leader_election.cc:288] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc593831e0704dbb8979dcc4548fadc3 (127.2.164.67:46873), c79d38c757e54ab5bd6bd592dc5f1ff8 (127.2.164.65:43577)
I20260926 19:43:55.945641 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.939598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:36259 (local address 127.2.164.67:46873)
0926 19:43:55.939812 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:55.939820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:55.940504 (+ 684us) server_negotiation.cc:407] Connection header received
0926 19:43:55.940655 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:55.940659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:55.940717 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:55.940797 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:55.941453 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.942148 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.943861 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.944694 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.944828 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.944877 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:55.945283 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 19:43:55.945414 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 19:43:55.945641 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.939682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:37505 (local address 127.2.164.65:43577)
0926 19:43:55.939905 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:55.939912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:55.940456 (+ 544us) server_negotiation.cc:407] Connection header received
0926 19:43:55.940625 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:55.940628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:55.940698 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:55.940769 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:55.941379 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.942047 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.943831 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.944626 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.944761 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.944829 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:55.945223 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 19:43:55.945355 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 19:43:55.945624 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.939575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:46873 (local address 127.2.164.66:36259)
0926 19:43:55.940396 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:55.940410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:55.940529 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:55.940823 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:55.940827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:55.940849 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:55.941292 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.941304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.942285 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.942291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.943653 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.943665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.944954 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.944960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.945112 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.945140 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 19:43:55.945233 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 19:43:55.945326 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":425,"threads_started":1}
I20260926 19:43:55.945648 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:55.939399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43577 (local address 127.2.164.66:37505)
0926 19:43:55.940339 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:55.940354 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:55.940493 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:55.940797 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:55.940801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:55.940824 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:55.941237 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.941248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.942166 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.942171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.943642 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:55.943655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:55.944814 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:55.944820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:55.945001 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:55.945035 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:43:55.945129 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 19:43:55.945270 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":1,"thread_start_us":659,"threads_started":1}
I20260926 19:43:55.946781 21589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860b6c9baa12468da58cab8cfc0676f3" candidate_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" is_pre_election: true
I20260926 19:43:55.946782 21850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860b6c9baa12468da58cab8cfc0676f3" candidate_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc593831e0704dbb8979dcc4548fadc3" is_pre_election: true
I20260926 19:43:55.947343 21589 raft_consensus.cc:2454] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06b6ba84d3d9445ea4d1d48e12d74373 in term 0.
I20260926 19:43:55.947343 21850 raft_consensus.cc:2454] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06b6ba84d3d9445ea4d1d48e12d74373 in term 0.
I20260926 19:43:55.947995 21654 leader_election.cc:302] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b6ba84d3d9445ea4d1d48e12d74373, c79d38c757e54ab5bd6bd592dc5f1ff8; no voters:
I20260926 19:43:55.948367 21921 raft_consensus.cc:2781] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:43:55.948503 21921 raft_consensus.cc:482] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:43:55.948604 21921 raft_consensus.cc:3037] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:55.950816 21921 raft_consensus.cc:504] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc593831e0704dbb8979dcc4548fadc3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46873 } } peers { permanent_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43577 } } peers { permanent_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37703 } }
I20260926 19:43:55.951404 21921 leader_election.cc:288] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [CANDIDATE]: Term 1 election: Requested vote from peers cc593831e0704dbb8979dcc4548fadc3 (127.2.164.67:46873), c79d38c757e54ab5bd6bd592dc5f1ff8 (127.2.164.65:43577)
I20260926 19:43:55.951853 21850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860b6c9baa12468da58cab8cfc0676f3" candidate_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc593831e0704dbb8979dcc4548fadc3"
I20260926 19:43:55.951925 21589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860b6c9baa12468da58cab8cfc0676f3" candidate_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8"
I20260926 19:43:55.952030 21850 raft_consensus.cc:3037] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:55.952121 21589 raft_consensus.cc:3037] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:55.954308 21850 raft_consensus.cc:2454] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06b6ba84d3d9445ea4d1d48e12d74373 in term 1.
I20260926 19:43:55.954300 21589 raft_consensus.cc:2454] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06b6ba84d3d9445ea4d1d48e12d74373 in term 1.
I20260926 19:43:55.954668 21653 leader_election.cc:302] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b6ba84d3d9445ea4d1d48e12d74373, cc593831e0704dbb8979dcc4548fadc3; no voters:
I20260926 19:43:55.954958 21921 raft_consensus.cc:2781] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:43:55.955399 21921 raft_consensus.cc:686] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 1 LEADER]: Becoming Leader. State: Replica: 06b6ba84d3d9445ea4d1d48e12d74373, State: Running, Role: LEADER
I20260926 19:43:55.955839 21921 consensus_queue.cc:237] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc593831e0704dbb8979dcc4548fadc3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46873 } } peers { permanent_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43577 } } peers { permanent_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37703 } }
I20260926 19:43:55.960311 21448 catalog_manager.cc:5495] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06b6ba84d3d9445ea4d1d48e12d74373 (127.2.164.66). New cstate: current_term: 1 leader_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc593831e0704dbb8979dcc4548fadc3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06b6ba84d3d9445ea4d1d48e12d74373" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 37703 } health_report { overall_health: HEALTHY } } }
W20260926 19:43:55.977118 21896 tablet.cc:2369] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:43:56.019915 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:56.014776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.2.164.66:37703)
0926 19:43:56.014972 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:56.014977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:56.014996 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:43:56.015093 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:56.015097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:56.015178 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:56.015307 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:43:56.015663 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:56.016366 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:56.017749 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:56.018157 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:56.018263 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:56.018822 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:43:56.019308 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:43:56.019469 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:56.019639 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 19:43:56.019681 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:43:56.027946 21850 raft_consensus.cc:1261] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3 [term 1 FOLLOWER]: Refusing update from remote peer 06b6ba84d3d9445ea4d1d48e12d74373: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:56.027946 21589 raft_consensus.cc:1261] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8 [term 1 FOLLOWER]: Refusing update from remote peer 06b6ba84d3d9445ea4d1d48e12d74373: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:56.028961 21924 consensus_queue.cc:1035] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc593831e0704dbb8979dcc4548fadc3" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:43:56.029291 21921 consensus_queue.cc:1035] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c79d38c757e54ab5bd6bd592dc5f1ff8" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 19:43:56.037374 21766 tablet.cc:2369] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:43:56.040177 21927 mvcc.cc:205] Tried to move back new op lower bound from 7333690720360550400 to 7333690720078512128. Current Snapshot: MvccSnapshot[applied={T|T < 7333690720360550400}]
I20260926 19:43:56.056526 21448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:56.056766 21448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:56.060074 21448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9acb249beac243e6933bc11175705ec0: Sending DeleteTablet for 3 replicas of tablet 860b6c9baa12468da58cab8cfc0676f3
I20260926 19:43:56.061486 21830 tablet_service.cc:1514] Processing DeleteTablet for tablet 860b6c9baa12468da58cab8cfc0676f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:56 UTC) from {username='slave'} at 127.0.0.1:44712
I20260926 19:43:56.061465 21569 tablet_service.cc:1514] Processing DeleteTablet for tablet 860b6c9baa12468da58cab8cfc0676f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:56 UTC) from {username='slave'} at 127.0.0.1:43620
I20260926 19:43:56.061486 21700 tablet_service.cc:1514] Processing DeleteTablet for tablet 860b6c9baa12468da58cab8cfc0676f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:56 UTC) from {username='slave'} at 127.0.0.1:37122
I20260926 19:43:56.063269 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 21507
I20260926 19:43:56.097093 21940 tablet_replica.cc:330] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: stopping tablet replica
I20260926 19:43:56.097563 21940 raft_consensus.cc:2229] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:43:56.098131 21940 raft_consensus.cc:2258] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:56.098838 21939 tablet_replica.cc:330] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: stopping tablet replica
I20260926 19:43:56.099164 21939 raft_consensus.cc:2229] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:56.099586 21939 raft_consensus.cc:2258] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:56.102226 21939 ts_tablet_manager.cc:1905] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:56.105947 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:56.064666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.2.164.65:43577)
0926 19:43:56.064868 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:56.064874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:56.089107 (+ 24233us) server_negotiation.cc:407] Connection header received
0926 19:43:56.089664 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:56.089671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:56.089808 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:56.089892 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:56.090646 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:56.091895 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:56.094049 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:56.094751 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:56.094846 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:56.100417 (+ 5571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:56.100428 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:56.100445 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:56.101209 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:56.101283 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:56.101311 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:56.101415 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:56.105442 (+ 4027us) server_negotiation.cc:299] Negotiation successful
0926 19:43:56.105558 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:43:56.106745 21940 ts_tablet_manager.cc:1905] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:56.107115 21629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:56.109318 21939 ts_tablet_manager.cc:1918] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:56.109465 21939 log.cc:1196] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/860b6c9baa12468da58cab8cfc0676f3
I20260926 19:43:56.109959 21939 ts_tablet_manager.cc:1939] T 860b6c9baa12468da58cab8cfc0676f3 P cc593831e0704dbb8979dcc4548fadc3: Deleting consensus metadata
I20260926 19:43:56.111691 21432 catalog_manager.cc:4841] TS cc593831e0704dbb8979dcc4548fadc3 (127.2.164.67:46873): tablet 860b6c9baa12468da58cab8cfc0676f3 (table test-table [id=223ab38186af49ed9946f14a435146f4]) successfully deleted
I20260926 19:43:56.113687 21940 ts_tablet_manager.cc:1918] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:56.113826 21940 log.cc:1196] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/860b6c9baa12468da58cab8cfc0676f3
I20260926 19:43:56.114216 21940 ts_tablet_manager.cc:1939] T 860b6c9baa12468da58cab8cfc0676f3 P 06b6ba84d3d9445ea4d1d48e12d74373: Deleting consensus metadata
I20260926 19:43:56.115542 21433 catalog_manager.cc:4841] TS 06b6ba84d3d9445ea4d1d48e12d74373 (127.2.164.66:37703): tablet 860b6c9baa12468da58cab8cfc0676f3 (table test-table [id=223ab38186af49ed9946f14a435146f4]) successfully deleted
I20260926 19:43:56.200848 21938 tablet_replica.cc:330] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8: stopping tablet replica
I20260926 19:43:56.201465 21938 raft_consensus.cc:2229] T 860b6c9baa12468da58cab8cfc0676f3 P c79d38c757e54ab5bd6bd592dc5f1ff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:56.201575 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 21507
W20260926 19:43:56.216073 21433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.164.65:43577 (error 108)
I20260926 19:43:56.217357 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 21638
I20260926 19:43:56.218016 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:56.217505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43577 (local address 127.0.0.1:43632)
0926 19:43:56.217720 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:56.217846 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 19:43:56.218423 21433 catalog_manager.cc:4584] TS c79d38c757e54ab5bd6bd592dc5f1ff8 (127.2.164.65:43577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 860b6c9baa12468da58cab8cfc0676f3: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43577: connect: Connection refused (error 111)
I20260926 19:43:56.225201 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:56.218152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.2.164.66:37703)
0926 19:43:56.218337 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:56.218343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:56.218399 (+ 56us) server_negotiation.cc:407] Connection header received
0926 19:43:56.218615 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:56.218621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:56.218728 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:56.218841 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:56.219183 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:56.220421 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:56.221737 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:56.222417 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:56.222541 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:56.223552 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:56.223566 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:56.223584 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:56.224422 (+ 838us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:56.224483 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:56.224509 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:56.224622 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:56.224708 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 19:43:56.224814 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:43:56.225939 21760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:56.271934 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:56.271466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43577 (local address 127.0.0.1:43636)
0926 19:43:56.271676 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:56.271771 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 19:43:56.273001 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:56.272663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43577 (local address 127.0.0.1:43646)
0926 19:43:56.272811 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:56.272882 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 19:43:56.333568 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:56.333083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43577 (local address 127.0.0.1:43652)
0926 19:43:56.333306 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:56.333431 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 19:43:56.334559 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:56.334160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43577 (local address 127.0.0.1:43664)
0926 19:43:56.334321 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:56.334430 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 19:43:56.335801 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 21638
I20260926 19:43:56.350541 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 21768
I20260926 19:43:56.357055 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:56.351247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44720 (local address 127.2.164.67:46873)
0926 19:43:56.351447 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:56.351454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:56.351511 (+ 57us) server_negotiation.cc:407] Connection header received
0926 19:43:56.351708 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:56.351712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:56.351793 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:56.351890 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:56.352287 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:56.352983 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:56.354328 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:56.354706 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:56.354836 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:56.355799 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:56.355808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:56.355824 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:43:56.356363 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:56.356421 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:43:56.356440 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:56.356543 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:56.356667 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 19:43:56.356771 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 19:43:56.357672 21890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:56.415658 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:56.415177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43577 (local address 127.0.0.1:43672)
0926 19:43:56.415395 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:56.415488 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 19:43:56.416746 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:56.416354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:43577 (local address 127.0.0.1:43674)
0926 19:43:56.416568 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:56.416640 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.65:43577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 19:43:56.444389 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 21768
I20260926 19:43:56.457525 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 21417
I20260926 19:43:56.458572 21478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:43:56.564293 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 21417
2026-09-26T19:43:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2706 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:43:56 dist-test-slave-mrw4 krb5kdc[21949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:43:56 dist-test-slave-mrw4 krb5kdc[21949](info): set up 2 sockets
Sep 26 19:43:56 dist-test-slave-mrw4 krb5kdc[21949](info): commencing operation
krb5kdc: starting...
W20260926 19:43:57.569094 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:43:57 dist-test-slave-mrw4 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:43:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:43:57.786383 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36871
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:36871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:58.094439 21962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:58.094703 21962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:58.094830 21962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:58.101876 21962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:43:58.101964 21962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:58.102041 21962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:58.102113 21962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:43:58.102169 21962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:43:58.111410 21962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:36871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:58.112555 21962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:58.113773 21962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:58.124279 21967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:58.124297 21970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:58.124447 21968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:58.125422 21962 server_base.cc:1034] running on GCE node
I20260926 19:43:58.126163 21962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:58.127437 21962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:58.128647 21962 hybrid_clock.cc:648] HybridClock initialized: now 1790451838128606 us; error 26 us; skew 500 ppm
I20260926 19:43:58.128974 21962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:58 dist-test-slave-mrw4 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451838, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:58.131875 21962 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:43:58.133282 21962 webserver.cc:460] Webserver started at http://127.2.164.126:36681/ using document root <none> and password file <none>
I20260926 19:43:58.133958 21962 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:58.134076 21962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:58.134356 21962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:58.137238 21962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "7e833fa50a3a466fb0e8ea9c0df8f871"
format_stamp: "Formatted at 2026-09-26 19:43:58 on dist-test-slave-mrw4"
server_key: "cd0b7787c858e53ac4d5f608fc4c237f"
server_key_iv: "f4d35a9c3bc59d2cc6acaf82da6277b6"
server_key_version: "encryptionkey@0"
I20260926 19:43:58.137982 21962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "7e833fa50a3a466fb0e8ea9c0df8f871"
format_stamp: "Formatted at 2026-09-26 19:43:58 on dist-test-slave-mrw4"
server_key: "cd0b7787c858e53ac4d5f608fc4c237f"
server_key_iv: "f4d35a9c3bc59d2cc6acaf82da6277b6"
server_key_version: "encryptionkey@0"
I20260926 19:43:58.143050 21962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 19:43:58.146287 21977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:58.147485 21962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260926 19:43:58.147636 21962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "7e833fa50a3a466fb0e8ea9c0df8f871"
format_stamp: "Formatted at 2026-09-26 19:43:58 on dist-test-slave-mrw4"
server_key: "cd0b7787c858e53ac4d5f608fc4c237f"
server_key_iv: "f4d35a9c3bc59d2cc6acaf82da6277b6"
server_key_version: "encryptionkey@0"
I20260926 19:43:58.147799 21962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:58.161718 21962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:58.165603 21962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:58.165841 21962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:58.181885 21962 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:36871
I20260926 19:43:58.181913 22028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:36871 every 8 connection(s)
I20260926 19:43:58.183084 21962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:43:58.184640 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 21962
I20260926 19:43:58.184800 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:43:58.185230 2705 external_mini_cluster.cc:1428] Setting key e7215dade272cf10eeffdc22d6660955
I20260926 19:43:58.186914 22029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:43:58 dist-test-slave-mrw4 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:58.199445 22029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871: Bootstrap starting.
I20260926 19:43:58.203037 22029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:58.204273 22029 log.cc:824] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:58.206959 22029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871: No bootstrap required, opened a new log
I20260926 19:43:58.208185 22032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:58.186871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.2.164.126:36871)
0926 19:43:58.187624 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:58.187639 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:58.187683 (+ 44us) server_negotiation.cc:407] Connection header received
0926 19:43:58.189759 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:58.189784 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:58.190267 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:58.190674 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:58.192100 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:58.194094 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:58.195387 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:58.196344 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:58.196520 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:58.200570 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:58.200580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:58.200597 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:58.200645 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:58.204346 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:58.205001 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:58.205009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:58.205016 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:58.205248 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:58.205617 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:58.205621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:58.205623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:58.206124 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:58.206387 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:58.206843 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 19:43:58.207264 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":305,"threads_started":1}
I20260926 19:43:58.214758 22029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e833fa50a3a466fb0e8ea9c0df8f871" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36871 } }
I20260926 19:43:58.215077 22029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:58.215161 22029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e833fa50a3a466fb0e8ea9c0df8f871, State: Initialized, Role: FOLLOWER
I20260926 19:43:58.215669 22029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e833fa50a3a466fb0e8ea9c0df8f871" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36871 } }
I20260926 19:43:58.215893 22029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:43:58.216010 22029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:43:58.216179 22029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:58.218218 22029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e833fa50a3a466fb0e8ea9c0df8f871" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36871 } }
I20260926 19:43:58.218658 22029 leader_election.cc:302] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e833fa50a3a466fb0e8ea9c0df8f871; no voters:
I20260926 19:43:58.219178 22029 leader_election.cc:288] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:43:58.219276 22034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:43:58.220232 22034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [term 1 LEADER]: Becoming Leader. State: Replica: 7e833fa50a3a466fb0e8ea9c0df8f871, State: Running, Role: LEADER
I20260926 19:43:58.220748 22034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e833fa50a3a466fb0e8ea9c0df8f871" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36871 } }
I20260926 19:43:58.221050 22029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:43:58.223444 22036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e833fa50a3a466fb0e8ea9c0df8f871. Latest consensus state: current_term: 1 leader_uuid: "7e833fa50a3a466fb0e8ea9c0df8f871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e833fa50a3a466fb0e8ea9c0df8f871" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36871 } } }
I20260926 19:43:58.223853 22036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:58.223467 22035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e833fa50a3a466fb0e8ea9c0df8f871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e833fa50a3a466fb0e8ea9c0df8f871" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36871 } } }
I20260926 19:43:58.224370 22035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871 [sys.catalog]: This master's current role is: LEADER
I20260926 19:43:58.225871 22043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:43:58.230813 22043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:43:58.239094 22043 catalog_manager.cc:1347] Generated new cluster ID: e98ccf6138754fda9d3fedf0cc104914
I20260926 19:43:58.239207 22043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:43:58.251833 22043 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:43:58.253120 22043 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:43:58.267377 22043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871: Generated new TSK 0
I20260926 19:43:58.268390 22043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.164.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
Entry for principal kudu/127.2.164.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab.
I20260926 19:43:58.328358 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36871
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:58.635594 22055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:58.635843 22055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:58.635933 22055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:58.642796 22055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:58.642889 22055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:58.643000 22055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:43:58.652217 22055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.65.keytab
--principal=kudu/127.2.164.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:58.653367 22055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:58.654538 22055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:58.666500 22060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:58.666492 22061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:58.666517 22063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:58.667309 22055 server_base.cc:1034] running on GCE node
I20260926 19:43:58.667862 22055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:58.668718 22055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:58.669914 22055 hybrid_clock.cc:648] HybridClock initialized: now 1790451838669856 us; error 48 us; skew 500 ppm
I20260926 19:43:58.670197 22055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:58 dist-test-slave-mrw4 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451838, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:58.677959 22055 init.cc:377] Logged in from keytab as kudu/127.2.164.65@KRBTEST.COM (short username kudu)
I20260926 19:43:58.679332 22055 webserver.cc:460] Webserver started at http://127.2.164.65:44077/ using document root <none> and password file <none>
I20260926 19:43:58.680047 22055 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:58.680143 22055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:58.680389 22055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:58.683073 22055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "e48efe1d985a4d87bd12ef97592baf81"
format_stamp: "Formatted at 2026-09-26 19:43:58 on dist-test-slave-mrw4"
server_key: "d6f12ac5b1870ac1abbef3d012a079a6"
server_key_iv: "0dc50a1edc1437368ea403958d6e902e"
server_key_version: "encryptionkey@0"
I20260926 19:43:58.683741 22055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "e48efe1d985a4d87bd12ef97592baf81"
format_stamp: "Formatted at 2026-09-26 19:43:58 on dist-test-slave-mrw4"
server_key: "d6f12ac5b1870ac1abbef3d012a079a6"
server_key_iv: "0dc50a1edc1437368ea403958d6e902e"
server_key_version: "encryptionkey@0"
I20260926 19:43:58.688405 22055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:43:58.691569 22070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:58.692847 22055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:43:58.692971 22055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "e48efe1d985a4d87bd12ef97592baf81"
format_stamp: "Formatted at 2026-09-26 19:43:58 on dist-test-slave-mrw4"
server_key: "d6f12ac5b1870ac1abbef3d012a079a6"
server_key_iv: "0dc50a1edc1437368ea403958d6e902e"
server_key_version: "encryptionkey@0"
I20260926 19:43:58.693128 22055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:58.707578 22055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:58.711263 22055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:58.711529 22055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:58.712368 22055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:58.713619 22055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:58.713694 22055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:58.713814 22055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:58.713868 22055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:58.742616 22055 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:44175
I20260926 19:43:58.742652 22182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:44175 every 8 connection(s)
I20260926 19:43:58.743790 22055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:43:58.748457 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 22055
I20260926 19:43:58.748636 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:43:58.749207 2705 external_mini_cluster.cc:1428] Setting key fcdb00ef9bad20eb8194d9fa388a538c
Sep 26 19:43:58 dist-test-slave-mrw4 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451838, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.65@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:58.764022 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:58.746213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:44795 (local address 127.2.164.126:36871)
0926 19:43:58.746767 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:58.746771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:58.747680 (+ 909us) server_negotiation.cc:407] Connection header received
0926 19:43:58.750028 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:58.750033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:58.750113 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:58.750214 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:58.751893 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:58.752941 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:58.754502 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:58.755025 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:58.755141 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:58.758655 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:58.758660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:58.758663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:58.758696 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:58.761127 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:58.761812 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:58.761816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:58.761818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:58.761943 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:58.762351 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:58.762355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:58.762357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:58.762631 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:58.762765 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:58.763430 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 19:43:58.763651 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":285,"threads_started":1}
I20260926 19:43:58.764901 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:58.746817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36871 (local address 127.2.164.65:44795)
0926 19:43:58.747529 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:58.747571 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:58.749762 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:58.750461 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:58.750469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:58.751087 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:58.751657 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:58.751675 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:58.753103 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:58.753107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:58.754346 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:58.754357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:58.755227 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:58.755232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:58.755451 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:58.756042 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:58.756062 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:58.758439 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:58.761269 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:58.761278 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:58.761294 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:58.761688 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:58.762044 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:58.762050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:58.762053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:58.762230 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:58.762753 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:58.762760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:58.763077 (+ 317us) client_negotiation.cc:769] Sending connection context
0926 19:43:58.763304 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 19:43:58.763786 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":267,"threads_started":1}
I20260926 19:43:58.766475 22183 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36871
I20260926 19:43:58.766765 22183 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:58.767572 22183 heartbeater.cc:511] Master 127.2.164.126:36871 requested a full tablet report, sending...
I20260926 19:43:58.769781 21994 ts_manager.cc:194] Registered new tserver with Master: e48efe1d985a4d87bd12ef97592baf81 (127.2.164.65:44175)
I20260926 19:43:58.771193 21994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.65@KRBTEST.COM'} at 127.2.164.65:44795
WARNING: no policy specified for kudu/127.2.164.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
Entry for principal kudu/127.2.164.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab.
I20260926 19:43:58.799698 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36871
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:59.102823 22189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:59.103094 22189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:59.103214 22189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:59.110195 22189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:59.110303 22189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:59.110457 22189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:43:59.119513 22189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.66.keytab
--principal=kudu/127.2.164.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:59.120694 22189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:59.121965 22189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:59.134080 22197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:59.134045 22194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:59.134126 22195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:59.134642 22189 server_base.cc:1034] running on GCE node
I20260926 19:43:59.135232 22189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:59.136086 22189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:59.137284 22189 hybrid_clock.cc:648] HybridClock initialized: now 1790451839137241 us; error 29 us; skew 500 ppm
I20260926 19:43:59.137571 22189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:59 dist-test-slave-mrw4 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:59.140491 22189 init.cc:377] Logged in from keytab as kudu/127.2.164.66@KRBTEST.COM (short username kudu)
I20260926 19:43:59.141822 22189 webserver.cc:460] Webserver started at http://127.2.164.66:32813/ using document root <none> and password file <none>
I20260926 19:43:59.142438 22189 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:59.142547 22189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:59.142822 22189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:59.145525 22189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "f9efd70b5ba3445199d952618206fd70"
format_stamp: "Formatted at 2026-09-26 19:43:59 on dist-test-slave-mrw4"
server_key: "8e9b566644c17ea3e49319236b56df92"
server_key_iv: "4c5c5382d1c7f5788a1dc3d153d691c0"
server_key_version: "encryptionkey@0"
I20260926 19:43:59.146227 22189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "f9efd70b5ba3445199d952618206fd70"
format_stamp: "Formatted at 2026-09-26 19:43:59 on dist-test-slave-mrw4"
server_key: "8e9b566644c17ea3e49319236b56df92"
server_key_iv: "4c5c5382d1c7f5788a1dc3d153d691c0"
server_key_version: "encryptionkey@0"
I20260926 19:43:59.150945 22189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:43:59.154247 22204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:59.155475 22189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:43:59.155629 22189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "f9efd70b5ba3445199d952618206fd70"
format_stamp: "Formatted at 2026-09-26 19:43:59 on dist-test-slave-mrw4"
server_key: "8e9b566644c17ea3e49319236b56df92"
server_key_iv: "4c5c5382d1c7f5788a1dc3d153d691c0"
server_key_version: "encryptionkey@0"
I20260926 19:43:59.155814 22189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:59.178287 22189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:59.182134 22189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:59.182387 22189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:59.183274 22189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:59.184588 22189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:59.184664 22189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:59.184767 22189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:59.184823 22189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:59.216419 22189 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:45657
I20260926 19:43:59.216468 22316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:45657 every 8 connection(s)
I20260926 19:43:59.217667 22189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:43:59.218804 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 22189
I20260926 19:43:59.218969 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:43:59.219511 2705 external_mini_cluster.cc:1428] Setting key a4b17c4c6eeb5489ceb93309417cf5b8
Sep 26 19:43:59 dist-test-slave-mrw4 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.66@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:59.237012 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.220762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:58151 (local address 127.2.164.126:36871)
0926 19:43:59.220941 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:59.220947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:59.222157 (+ 1210us) server_negotiation.cc:407] Connection header received
0926 19:43:59.223950 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:59.223955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:59.224040 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:59.224124 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:59.225728 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.226443 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.227916 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.228490 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.228610 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.231921 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:59.231926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:59.231930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:59.231960 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:59.234289 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:59.234950 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:59.234954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:59.234956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:59.235075 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:59.235459 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:59.235463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:59.235465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:59.235705 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:59.235840 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:59.236475 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 19:43:59.236688 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:43:59.237937 22319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.221301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36871 (local address 127.2.164.66:58151)
0926 19:43:59.222023 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:59.222058 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:59.223706 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:59.224387 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:59.224398 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:59.224940 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:59.225560 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.225578 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.226578 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.226584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.227783 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.227794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.228690 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.228695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.228889 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.229518 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:59.229538 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:59.231729 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:59.234416 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:59.234425 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:59.234440 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:59.234835 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:59.235171 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:59.235176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:59.235180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:59.235353 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:59.235825 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:59.235840 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:59.236133 (+ 293us) client_negotiation.cc:769] Sending connection context
0926 19:43:59.236348 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 19:43:59.236823 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":284,"threads_started":1}
I20260926 19:43:59.239259 22317 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36871
I20260926 19:43:59.239593 22317 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:59.240423 22317 heartbeater.cc:511] Master 127.2.164.126:36871 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.164.67@KRBTEST.COM; defaulting to no policy
I20260926 19:43:59.241919 21994 ts_manager.cc:194] Registered new tserver with Master: f9efd70b5ba3445199d952618206fd70 (127.2.164.66:45657)
I20260926 19:43:59.242684 21994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.66@KRBTEST.COM'} at 127.2.164.66:58151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
Entry for principal kudu/127.2.164.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab.
I20260926 19:43:59.268404 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36871
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:43:59.577101 22322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:43:59.577385 22322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:43:59.577523 22322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:43:59.584395 22322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:43:59.584507 22322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:43:59.584651 22322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:43:59.594125 22322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.67.keytab
--principal=kudu/127.2.164.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:36871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:43:59.595374 22322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:43:59.596735 22322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:43:59.609088 22327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:59.609071 22328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:43:59.609138 22330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:43:59.610148 22322 server_base.cc:1034] running on GCE node
I20260926 19:43:59.610702 22322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:43:59.611567 22322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:43:59.612771 22322 hybrid_clock.cc:648] HybridClock initialized: now 1790451839612719 us; error 43 us; skew 500 ppm
I20260926 19:43:59.613063 22322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:43:59 dist-test-slave-mrw4 krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:43:59.615919 22322 init.cc:377] Logged in from keytab as kudu/127.2.164.67@KRBTEST.COM (short username kudu)
I20260926 19:43:59.617324 22322 webserver.cc:460] Webserver started at http://127.2.164.67:33033/ using document root <none> and password file <none>
I20260926 19:43:59.617946 22322 fs_manager.cc:359] Metadata directory not provided
I20260926 19:43:59.618057 22322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:43:59.618321 22322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:43:59.620988 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "3c905910c69b46bc8e64bb7e691db024"
format_stamp: "Formatted at 2026-09-26 19:43:59 on dist-test-slave-mrw4"
server_key: "a2f0be55f1eca368d963038128d2d711"
server_key_iv: "1a778806013479d3dd9f4391dcc6956d"
server_key_version: "encryptionkey@0"
I20260926 19:43:59.621778 22322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "3c905910c69b46bc8e64bb7e691db024"
format_stamp: "Formatted at 2026-09-26 19:43:59 on dist-test-slave-mrw4"
server_key: "a2f0be55f1eca368d963038128d2d711"
server_key_iv: "1a778806013479d3dd9f4391dcc6956d"
server_key_version: "encryptionkey@0"
I20260926 19:43:59.626468 22322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:43:59.629706 22337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:59.631029 22322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:43:59.631178 22322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "3c905910c69b46bc8e64bb7e691db024"
format_stamp: "Formatted at 2026-09-26 19:43:59 on dist-test-slave-mrw4"
server_key: "a2f0be55f1eca368d963038128d2d711"
server_key_iv: "1a778806013479d3dd9f4391dcc6956d"
server_key_version: "encryptionkey@0"
I20260926 19:43:59.631349 22322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:43:59.662531 22322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:43:59.666464 22322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:43:59.666710 22322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:43:59.667605 22322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:43:59.668855 22322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:43:59.668938 22322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:59.669049 22322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:43:59.669106 22322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:43:59.698993 22322 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:42257
I20260926 19:43:59.699025 22449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:42257 every 8 connection(s)
I20260926 19:43:59.700217 22322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:43:59.708593 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 22322
I20260926 19:43:59.708753 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:43:59.709156 2705 external_mini_cluster.cc:1428] Setting key 88da947fdbc68942f34929ab02f8fd3b
Sep 26 19:43:59 dist-test-slave-mrw4 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451839, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.67@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:43:59.719966 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.702692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:58939 (local address 127.2.164.126:36871)
0926 19:43:59.702951 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:59.702957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:59.704225 (+ 1268us) server_negotiation.cc:407] Connection header received
0926 19:43:59.706616 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:59.706622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:59.706710 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:59.706800 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:59.708271 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.709115 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.710972 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.711403 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.711510 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.714761 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:59.714766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:59.714769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:59.714804 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:59.717056 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:59.717742 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:59.717747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:59.717751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:59.717873 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:59.718274 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:59.718278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:59.718280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:59.718551 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:59.718684 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:59.719397 (+ 713us) server_negotiation.cc:299] Negotiation successful
0926 19:43:59.719619 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 19:43:59.720865 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.703213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:36871 (local address 127.2.164.67:58939)
0926 19:43:59.704054 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:59.704107 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:59.706342 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:59.707053 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:59.707061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:59.707551 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:43:59.708104 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.708120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.709271 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.709277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.710850 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.710862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.711607 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.711614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.711868 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.712439 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:43:59.712461 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:43:59.714562 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:43:59.717203 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:59.717212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:59.717228 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:59.717629 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:59.717973 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:43:59.717978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:43:59.717982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:43:59.718159 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:43:59.718682 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:43:59.718690 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:43:59.719006 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 19:43:59.719287 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 19:43:59.719775 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":344,"threads_started":1}
I20260926 19:43:59.722093 22450 heartbeater.cc:347] Connected to a master server at 127.2.164.126:36871
I20260926 19:43:59.722400 22450 heartbeater.cc:464] Registering TS with master...
I20260926 19:43:59.723112 22450 heartbeater.cc:511] Master 127.2.164.126:36871 requested a full tablet report, sending...
I20260926 19:43:59.724335 21994 ts_manager.cc:194] Registered new tserver with Master: 3c905910c69b46bc8e64bb7e691db024 (127.2.164.67:42257)
I20260926 19:43:59.725188 21994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.164.67@KRBTEST.COM'} at 127.2.164.67:58939
I20260926 19:43:59.735581 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:43:59.753472 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.739408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.2.164.126:36871)
0926 19:43:59.739557 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:59.739563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:59.740007 (+ 444us) server_negotiation.cc:407] Connection header received
0926 19:43:59.740173 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:59.740178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:59.740249 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:59.740325 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:59.741712 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.742712 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.744062 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.744653 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.744774 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.747486 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:59.747493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:59.747497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:59.747535 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:59.750787 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:59.751454 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:59.751459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:59.751462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:59.751616 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:59.752008 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:59.752013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:59.752016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:59.752352 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:59.752575 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:59.752983 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 19:43:59.753227 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:43:59.756975 21994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:43:59.763553 21994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:43:59.773994 22183 heartbeater.cc:503] Master 127.2.164.126:36871 was elected leader, sending a full tablet report...
I20260926 19:43:59.801143 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.775690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.2.164.65:44175)
0926 19:43:59.792095 (+ 16405us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:59.792103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:59.792132 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:43:59.792231 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:59.792239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:59.792495 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:59.793579 (+ 1084us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:59.793808 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.795390 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.798507 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.799948 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.800250 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.800455 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:59.800656 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 19:43:59.800805 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16246,"thread_start_us":16924,"threads_started":1}
I20260926 19:43:59.801354 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.775094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44175 (local address 127.0.0.1:52500)
0926 19:43:59.776380 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:59.776425 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:59.776584 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:59.792884 (+ 16300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:59.792892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:59.792935 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:59.793376 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.793389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.795541 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.795546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.798006 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.798024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.800175 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.800181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.800422 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.800493 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 19:43:59.800629 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 19:43:59.800904 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"mutex_wait_us":125,"thread_start_us":18068,"threads_started":1}
I20260926 19:43:59.801723 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.775739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.2.164.66:45657)
0926 19:43:59.788266 (+ 12527us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:59.788276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:59.792463 (+ 4187us) server_negotiation.cc:407] Connection header received
0926 19:43:59.792562 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:59.792569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:59.792790 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:59.792963 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:59.794052 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.795541 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.798829 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.799792 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.799957 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.800163 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:59.801126 (+ 963us) server_negotiation.cc:299] Negotiation successful
0926 19:43:59.801321 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12363,"thread_start_us":12244,"threads_started":1}
I20260926 19:43:59.801934 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.775204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:45657 (local address 127.0.0.1:57162)
0926 19:43:59.792204 (+ 17000us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:59.792226 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:59.792337 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:59.793393 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:59.793397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:59.793434 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:59.793850 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.793863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.795676 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.795681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.798691 (+ 3010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.798704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.799986 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.799992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.800229 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.800804 (+ 575us) client_negotiation.cc:769] Sending connection context
0926 19:43:59.800879 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 19:43:59.800980 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16889,"mutex_wait_us":1417,"thread_start_us":833,"threads_started":1}
I20260926 19:43:59.805111 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.776822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:42257 (local address 127.0.0.1:58304)
0926 19:43:59.797242 (+ 20420us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:59.797259 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:59.797379 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:59.797929 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:59.797933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:59.797960 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:59.798439 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.798452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.800302 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.800307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.802431 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.802442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.804365 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.804371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.804527 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.804557 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 19:43:59.804652 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 19:43:59.804765 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20248,"mutex_wait_us":56,"thread_start_us":20103,"threads_started":1}
I20260926 19:43:59.805193 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.777203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58304 (local address 127.2.164.67:42257)
0926 19:43:59.796791 (+ 19588us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:59.796800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:59.797495 (+ 695us) server_negotiation.cc:407] Connection header received
0926 19:43:59.797590 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:59.797597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:59.797759 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:59.797918 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:59.798641 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.800165 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.802569 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.803917 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.804037 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.804238 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:59.804721 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 19:43:59.804871 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19412,"thread_start_us":19261,"threads_started":1}
I20260926 19:43:59.805382 22252 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9570bc003e450d93bc9ac39bc4236e (DEFAULT_TABLE table=test-table [id=c155f9f3cf314c37bbb051fcff431aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:59.805543 22118 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9570bc003e450d93bc9ac39bc4236e (DEFAULT_TABLE table=test-table [id=c155f9f3cf314c37bbb051fcff431aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:59.806720 22252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9570bc003e450d93bc9ac39bc4236e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:59.806816 22118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9570bc003e450d93bc9ac39bc4236e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:59.808672 22385 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9570bc003e450d93bc9ac39bc4236e (DEFAULT_TABLE table=test-table [id=c155f9f3cf314c37bbb051fcff431aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:43:59.809927 22385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9570bc003e450d93bc9ac39bc4236e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:43:59.817034 22468 tablet_bootstrap.cc:492] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81: Bootstrap starting.
I20260926 19:43:59.820606 22468 tablet_bootstrap.cc:654] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:59.821746 22468 log.cc:824] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:59.821823 22469 tablet_bootstrap.cc:492] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70: Bootstrap starting.
I20260926 19:43:59.823632 22471 tablet_bootstrap.cc:492] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024: Bootstrap starting.
I20260926 19:43:59.824766 22468 tablet_bootstrap.cc:492] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81: No bootstrap required, opened a new log
I20260926 19:43:59.825038 22468 ts_tablet_manager.cc:1397] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 19:43:59.825052 22469 tablet_bootstrap.cc:654] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:59.826275 22469 log.cc:824] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:59.826452 22471 tablet_bootstrap.cc:654] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024: Neither blocks nor log segments found. Creating new log.
I20260926 19:43:59.827548 22471 log.cc:824] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024: Log is configured to *not* fsync() on all Append() calls
I20260926 19:43:59.829515 22469 tablet_bootstrap.cc:492] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70: No bootstrap required, opened a new log
I20260926 19:43:59.829762 22469 ts_tablet_manager.cc:1397] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 19:43:59.830086 22471 tablet_bootstrap.cc:492] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024: No bootstrap required, opened a new log
I20260926 19:43:59.830299 22471 ts_tablet_manager.cc:1397] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 19:43:59.834718 22468 raft_consensus.cc:348] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48efe1d985a4d87bd12ef97592baf81" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44175 } } peers { permanent_uuid: "f9efd70b5ba3445199d952618206fd70" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45657 } } peers { permanent_uuid: "3c905910c69b46bc8e64bb7e691db024" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42257 } }
I20260926 19:43:59.835156 22468 raft_consensus.cc:374] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:59.835251 22468 raft_consensus.cc:729] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48efe1d985a4d87bd12ef97592baf81, State: Initialized, Role: FOLLOWER
I20260926 19:43:59.835934 22468 consensus_queue.cc:260] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48efe1d985a4d87bd12ef97592baf81" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44175 } } peers { permanent_uuid: "f9efd70b5ba3445199d952618206fd70" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45657 } } peers { permanent_uuid: "3c905910c69b46bc8e64bb7e691db024" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42257 } }
I20260926 19:43:59.837400 22471 raft_consensus.cc:348] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48efe1d985a4d87bd12ef97592baf81" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44175 } } peers { permanent_uuid: "f9efd70b5ba3445199d952618206fd70" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45657 } } peers { permanent_uuid: "3c905910c69b46bc8e64bb7e691db024" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42257 } }
I20260926 19:43:59.837751 22468 ts_tablet_manager.cc:1428] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 19:43:59.837883 22471 raft_consensus.cc:374] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:59.837998 22471 raft_consensus.cc:729] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c905910c69b46bc8e64bb7e691db024, State: Initialized, Role: FOLLOWER
I20260926 19:43:59.838582 22471 consensus_queue.cc:260] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48efe1d985a4d87bd12ef97592baf81" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44175 } } peers { permanent_uuid: "f9efd70b5ba3445199d952618206fd70" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45657 } } peers { permanent_uuid: "3c905910c69b46bc8e64bb7e691db024" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42257 } }
I20260926 19:43:59.839393 22469 raft_consensus.cc:348] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48efe1d985a4d87bd12ef97592baf81" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44175 } } peers { permanent_uuid: "f9efd70b5ba3445199d952618206fd70" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45657 } } peers { permanent_uuid: "3c905910c69b46bc8e64bb7e691db024" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42257 } }
I20260926 19:43:59.839777 22469 raft_consensus.cc:374] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:43:59.839830 22471 ts_tablet_manager.cc:1428] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260926 19:43:59.839877 22469 raft_consensus.cc:729] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9efd70b5ba3445199d952618206fd70, State: Initialized, Role: FOLLOWER
I20260926 19:43:59.840443 22469 consensus_queue.cc:260] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48efe1d985a4d87bd12ef97592baf81" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44175 } } peers { permanent_uuid: "f9efd70b5ba3445199d952618206fd70" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45657 } } peers { permanent_uuid: "3c905910c69b46bc8e64bb7e691db024" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42257 } }
I20260926 19:43:59.840785 22450 heartbeater.cc:503] Master 127.2.164.126:36871 was elected leader, sending a full tablet report...
I20260926 19:43:59.841866 22475 raft_consensus.cc:482] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:43:59.842088 22475 raft_consensus.cc:504] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48efe1d985a4d87bd12ef97592baf81" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44175 } } peers { permanent_uuid: "f9efd70b5ba3445199d952618206fd70" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45657 } } peers { permanent_uuid: "3c905910c69b46bc8e64bb7e691db024" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42257 } }
I20260926 19:43:59.843559 22475 leader_election.cc:288] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e48efe1d985a4d87bd12ef97592baf81 (127.2.164.65:44175), f9efd70b5ba3445199d952618206fd70 (127.2.164.66:45657)
I20260926 19:43:59.847532 22317 heartbeater.cc:503] Master 127.2.164.126:36871 was elected leader, sending a full tablet report...
I20260926 19:43:59.848130 22469 ts_tablet_manager.cc:1428] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260926 19:43:59.851181 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.843824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:44175 (local address 127.2.164.67:33605)
0926 19:43:59.844374 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:59.844394 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:59.844508 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:59.845298 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:59.845304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:59.845333 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:59.845558 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.845572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.847033 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.847038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.848958 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.848972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.850478 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.850484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.850663 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.850694 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:43:59.850772 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 19:43:59.850918 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405}
I20260926 19:43:59.851606 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.844444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:41271 (local address 127.2.164.66:45657)
0926 19:43:59.845215 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:59.845221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:59.845866 (+ 645us) server_negotiation.cc:407] Connection header received
0926 19:43:59.846002 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:59.846007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:59.846064 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:59.846133 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:59.846584 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.847479 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.849631 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.850678 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.850835 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.850898 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:59.851210 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 19:43:59.851375 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598}
I20260926 19:43:59.851818 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.844336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:33605 (local address 127.2.164.65:44175)
0926 19:43:59.844647 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:59.844654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:59.844674 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:43:59.844742 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:59.844748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:59.844833 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:59.844927 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:43:59.845754 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.846901 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.849098 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.850325 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.850487 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.850556 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:59.851456 (+ 900us) server_negotiation.cc:299] Negotiation successful
0926 19:43:59.851590 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260926 19:43:59.852207 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.844031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:45657 (local address 127.2.164.67:41271)
0926 19:43:59.845774 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0926 19:43:59.845790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:43:59.845896 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:43:59.846153 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:43:59.846157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:43:59.846178 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:43:59.846457 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.846466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.847652 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.847657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.849497 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:43:59.849507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.850881 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.850887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:43:59.851060 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.851091 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:43:59.851174 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 19:43:59.851462 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1642,"mutex_wait_us":1,"thread_start_us":1478,"threads_started":1}
I20260926 19:43:59.852623 22138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9570bc003e450d93bc9ac39bc4236e" candidate_uuid: "3c905910c69b46bc8e64bb7e691db024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48efe1d985a4d87bd12ef97592baf81" is_pre_election: true
I20260926 19:43:59.853044 22138 raft_consensus.cc:2454] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c905910c69b46bc8e64bb7e691db024 in term 0.
I20260926 19:43:59.853030 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9570bc003e450d93bc9ac39bc4236e" candidate_uuid: "3c905910c69b46bc8e64bb7e691db024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9efd70b5ba3445199d952618206fd70" is_pre_election: true
I20260926 19:43:59.853425 22272 raft_consensus.cc:2454] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c905910c69b46bc8e64bb7e691db024 in term 0.
I20260926 19:43:59.853634 22338 leader_election.cc:302] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c905910c69b46bc8e64bb7e691db024, e48efe1d985a4d87bd12ef97592baf81; no voters:
I20260926 19:43:59.853971 22475 raft_consensus.cc:2781] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:43:59.854089 22475 raft_consensus.cc:482] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:43:59.854185 22475 raft_consensus.cc:3037] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:59.856194 22475 raft_consensus.cc:504] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48efe1d985a4d87bd12ef97592baf81" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44175 } } peers { permanent_uuid: "f9efd70b5ba3445199d952618206fd70" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45657 } } peers { permanent_uuid: "3c905910c69b46bc8e64bb7e691db024" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42257 } }
I20260926 19:43:59.856732 22475 leader_election.cc:288] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [CANDIDATE]: Term 1 election: Requested vote from peers e48efe1d985a4d87bd12ef97592baf81 (127.2.164.65:44175), f9efd70b5ba3445199d952618206fd70 (127.2.164.66:45657)
I20260926 19:43:59.857092 22138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9570bc003e450d93bc9ac39bc4236e" candidate_uuid: "3c905910c69b46bc8e64bb7e691db024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48efe1d985a4d87bd12ef97592baf81"
I20260926 19:43:59.857157 22272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9570bc003e450d93bc9ac39bc4236e" candidate_uuid: "3c905910c69b46bc8e64bb7e691db024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9efd70b5ba3445199d952618206fd70"
I20260926 19:43:59.857290 22138 raft_consensus.cc:3037] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:59.857373 22272 raft_consensus.cc:3037] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:43:59.859380 22138 raft_consensus.cc:2454] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c905910c69b46bc8e64bb7e691db024 in term 1.
I20260926 19:43:59.859570 22272 raft_consensus.cc:2454] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c905910c69b46bc8e64bb7e691db024 in term 1.
I20260926 19:43:59.859706 22338 leader_election.cc:302] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c905910c69b46bc8e64bb7e691db024, e48efe1d985a4d87bd12ef97592baf81; no voters:
I20260926 19:43:59.859978 22475 raft_consensus.cc:2781] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:43:59.860515 22475 raft_consensus.cc:686] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 1 LEADER]: Becoming Leader. State: Replica: 3c905910c69b46bc8e64bb7e691db024, State: Running, Role: LEADER
I20260926 19:43:59.860976 22475 consensus_queue.cc:237] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48efe1d985a4d87bd12ef97592baf81" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44175 } } peers { permanent_uuid: "f9efd70b5ba3445199d952618206fd70" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45657 } } peers { permanent_uuid: "3c905910c69b46bc8e64bb7e691db024" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42257 } }
I20260926 19:43:59.864786 21993 catalog_manager.cc:5495] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c905910c69b46bc8e64bb7e691db024 (127.2.164.67). New cstate: current_term: 1 leader_uuid: "3c905910c69b46bc8e64bb7e691db024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48efe1d985a4d87bd12ef97592baf81" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9efd70b5ba3445199d952618206fd70" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c905910c69b46bc8e64bb7e691db024" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 42257 } health_report { overall_health: HEALTHY } } }
I20260926 19:43:59.890921 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.885800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.2.164.67:42257)
0926 19:43:59.886000 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:59.886006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:59.886108 (+ 102us) server_negotiation.cc:407] Connection header received
0926 19:43:59.886308 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:59.886312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:59.886392 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:59.886516 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:43:59.886868 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.887584 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.888905 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.889329 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.889444 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.889908 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:43:59.890373 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:43:59.890508 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:59.890606 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 19:43:59.890683 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:43:59.898427 22138 raft_consensus.cc:1261] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81 [term 1 FOLLOWER]: Refusing update from remote peer 3c905910c69b46bc8e64bb7e691db024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:59.898447 22272 raft_consensus.cc:1261] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70 [term 1 FOLLOWER]: Refusing update from remote peer 3c905910c69b46bc8e64bb7e691db024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:43:59.899384 22478 consensus_queue.cc:1035] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e48efe1d985a4d87bd12ef97592baf81" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:43:59.899688 22475 consensus_queue.cc:1035] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9efd70b5ba3445199d952618206fd70" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 45657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:43:59.911940 22481 mvcc.cc:205] Tried to move back new op lower bound from 7333690736214462464 to 7333690736074055680. Current Snapshot: MvccSnapshot[applied={T|T < 7333690736214462464}]
I20260926 19:43:59.924696 21993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:59.924985 21993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:43:59.928411 21993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e833fa50a3a466fb0e8ea9c0df8f871: Sending DeleteTablet for 3 replicas of tablet 3f9570bc003e450d93bc9ac39bc4236e
I20260926 19:43:59.929571 22252 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9570bc003e450d93bc9ac39bc4236e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:59 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:57162
I20260926 19:43:59.929579 22118 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9570bc003e450d93bc9ac39bc4236e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:59 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:52500
I20260926 19:43:59.929576 22385 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9570bc003e450d93bc9ac39bc4236e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:43:59 UTC) from {username='kudu', principal='kudu/127.2.164.126@KRBTEST.COM'} at 127.0.0.1:58304
I20260926 19:43:59.935089 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 22055
I20260926 19:43:59.939514 22492 tablet_replica.cc:330] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81: stopping tablet replica
I20260926 19:43:59.939953 22492 raft_consensus.cc:2229] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:59.940434 22492 raft_consensus.cc:2258] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:59.941463 22491 tablet_replica.cc:330] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024: stopping tablet replica
I20260926 19:43:59.941829 22491 raft_consensus.cc:2229] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:43:59.942438 22491 raft_consensus.cc:2258] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:59.943181 22490 tablet_replica.cc:330] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70: stopping tablet replica
I20260926 19:43:59.943521 22490 raft_consensus.cc:2229] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:43:59.943975 22490 raft_consensus.cc:2258] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:43:59.944622 22491 ts_tablet_manager.cc:1905] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:59.944985 22492 ts_tablet_manager.cc:1905] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:59.946080 22490 ts_tablet_manager.cc:1905] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:43:59.952100 22491 ts_tablet_manager.cc:1918] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:59.952378 22491 log.cc:1196] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/3f9570bc003e450d93bc9ac39bc4236e
I20260926 19:43:59.952664 22490 ts_tablet_manager.cc:1918] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:59.952814 22490 log.cc:1196] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/3f9570bc003e450d93bc9ac39bc4236e
I20260926 19:43:59.953131 22491 ts_tablet_manager.cc:1939] T 3f9570bc003e450d93bc9ac39bc4236e P 3c905910c69b46bc8e64bb7e691db024: Deleting consensus metadata
I20260926 19:43:59.953369 22490 ts_tablet_manager.cc:1939] T 3f9570bc003e450d93bc9ac39bc4236e P f9efd70b5ba3445199d952618206fd70: Deleting consensus metadata
I20260926 19:43:59.954988 21980 catalog_manager.cc:4841] TS f9efd70b5ba3445199d952618206fd70 (127.2.164.66:45657): tablet 3f9570bc003e450d93bc9ac39bc4236e (table test-table [id=c155f9f3cf314c37bbb051fcff431aa0]) successfully deleted
I20260926 19:43:59.955298 21980 catalog_manager.cc:4841] TS 3c905910c69b46bc8e64bb7e691db024 (127.2.164.67:42257): tablet 3f9570bc003e450d93bc9ac39bc4236e (table test-table [id=c155f9f3cf314c37bbb051fcff431aa0]) successfully deleted
I20260926 19:43:59.957269 22492 ts_tablet_manager.cc:1918] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:43:59.957434 22492 log.cc:1196] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/3f9570bc003e450d93bc9ac39bc4236e
I20260926 19:43:59.957892 22492 ts_tablet_manager.cc:1939] T 3f9570bc003e450d93bc9ac39bc4236e P e48efe1d985a4d87bd12ef97592baf81: Deleting consensus metadata
I20260926 19:43:59.959200 21978 catalog_manager.cc:4841] TS e48efe1d985a4d87bd12ef97592baf81 (127.2.164.65:44175): tablet 3f9570bc003e450d93bc9ac39bc4236e (table test-table [id=c155f9f3cf314c37bbb051fcff431aa0]) successfully deleted
Sep 26 19:43:59 dist-test-slave-mrw4 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.65@KRBTEST.COM
I20260926 19:43:59.969506 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:43:59.936325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.2.164.65:44175)
0926 19:43:59.936528 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 19:43:59.936535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:43:59.952790 (+ 16255us) server_negotiation.cc:407] Connection header received
0926 19:43:59.953298 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:43:59.953305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:43:59.953390 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:43:59.953483 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:43:59.955203 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.956415 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.958180 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:43:59.958831 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:43:59.958979 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:43:59.963307 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:43:59.963322 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:43:59.963341 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:43:59.963380 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:43:59.966822 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:59.967481 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:59.967490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:59.967493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:59.967644 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:43:59.967952 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:43:59.967956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:43:59.967960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:43:59.968411 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:43:59.968723 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 19:43:59.968893 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 19:43:59.969106 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 19:43:59.970137 22178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:00.057562 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 22055
I20260926 19:44:00.072582 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 22189
Sep 26 19:44:00 dist-test-slave-mrw4 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.66@KRBTEST.COM
I20260926 19:44:00.087695 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:00.073403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.2.164.66:45657)
0926 19:44:00.073576 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:00.073582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:00.073602 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:44:00.073691 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:00.073697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:00.073761 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:00.073853 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:00.075079 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:00.075826 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:00.077063 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:00.077537 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:00.077642 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:00.081984 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:00.081999 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:00.082015 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:00.082048 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:00.084645 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:00.085558 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:00.085566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:00.085568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:00.085699 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:00.086115 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:00.086119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:00.086121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:00.086529 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:00.086927 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:00.087170 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 19:44:00.087386 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:44:00.088392 22312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:00.175436 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 22189
I20260926 19:44:00.189438 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 22322
Sep 26 19:44:00 dist-test-slave-mrw4 krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.67@KRBTEST.COM
I20260926 19:44:00.205030 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:00.190129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.2.164.67:42257)
0926 19:44:00.190622 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:00.190626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:00.190641 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:44:00.190689 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:00.190692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:00.190774 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:00.190856 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:00.192164 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:00.192977 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:00.194325 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:00.194821 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:00.194939 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:00.199072 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:00.199083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:00.199099 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:00.199136 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:00.202142 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:00.202763 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:00.202771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:00.202773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:00.202933 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:00.203278 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:00.203281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:00.203283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:00.203621 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:00.203953 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:00.204164 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 19:44:00.204376 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387}
I20260926 19:44:00.205703 22445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:00.324666 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 22322
I20260926 19:44:00.340423 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 21962
I20260926 19:44:00.341447 22024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:00.449832 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 21962
2026-09-26T19:44:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3886 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T19:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:00Z Disabled control of system clock
I20260926 19:44:00.516111 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:43795
--webserver_interface=127.2.164.126
--webserver_port=0
--builtin_ntp_servers=127.2.164.84:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:43795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:44:00.824656 22501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:00.824908 22501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:00.825006 22501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:00.831909 22501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:00.832005 22501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:00.832063 22501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:00.832144 22501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:00.832198 22501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:00.841038 22501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:43795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:43795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:00.842235 22501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:00.843935 22501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:00.854763 22507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:00.854718 22509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:00.854795 22506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:00.855551 22501 server_base.cc:1034] running on GCE node
I20260926 19:44:00.856204 22501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:00.857509 22501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:00.858714 22501 hybrid_clock.cc:648] HybridClock initialized: now 1790451840858663 us; error 35 us; skew 500 ppm
I20260926 19:44:00.859018 22501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:44:00.861018 22501 webserver.cc:460] Webserver started at http://127.2.164.126:43635/ using document root <none> and password file <none>
I20260926 19:44:00.861701 22501 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:00.861797 22501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:00.862072 22501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:00.864789 22501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "8598a57db42646fcb57a566cb249e8b8"
format_stamp: "Formatted at 2026-09-26 19:44:00 on dist-test-slave-mrw4"
server_key: "76e34df7486e5396ea103b4b78192e9f"
server_key_iv: "29857169472d7b0c28a6bc28c4305a96"
server_key_version: "encryptionkey@0"
I20260926 19:44:00.865554 22501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "8598a57db42646fcb57a566cb249e8b8"
format_stamp: "Formatted at 2026-09-26 19:44:00 on dist-test-slave-mrw4"
server_key: "76e34df7486e5396ea103b4b78192e9f"
server_key_iv: "29857169472d7b0c28a6bc28c4305a96"
server_key_version: "encryptionkey@0"
I20260926 19:44:00.870157 22501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:44:00.873363 22515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:00.874558 22501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:44:00.874709 22501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "8598a57db42646fcb57a566cb249e8b8"
format_stamp: "Formatted at 2026-09-26 19:44:00 on dist-test-slave-mrw4"
server_key: "76e34df7486e5396ea103b4b78192e9f"
server_key_iv: "29857169472d7b0c28a6bc28c4305a96"
server_key_version: "encryptionkey@0"
I20260926 19:44:00.874890 22501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:00.913112 22501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:00.913911 22501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:00.914141 22501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:00.930444 22501 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:43795
I20260926 19:44:00.930462 22566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:43795 every 8 connection(s)
I20260926 19:44:00.931694 22501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:00.934940 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 22501
I20260926 19:44:00.935094 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:00.935210 22567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:00.935530 2705 external_mini_cluster.cc:1428] Setting key 5cc967dd624479bcc03a1161523304b5
I20260926 19:44:00.949646 22567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8: Bootstrap starting.
I20260926 19:44:00.949746 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:00.937310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.2.164.126:43795)
0926 19:44:00.938355 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:00.938369 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:00.938417 (+ 48us) server_negotiation.cc:407] Connection header received
0926 19:44:00.939966 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:00.940000 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:00.940615 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:00.941055 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:00.941609 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:00.943533 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:00.944840 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:00.945867 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:00.946051 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:00.947034 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:00.947044 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:00.947065 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:00.947765 (+ 700us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:00.947872 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:00.947896 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:00.948000 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:00.948190 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 19:44:00.948532 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":331,"threads_started":1}
I20260926 19:44:00.952682 22567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:00.953614 22567 log.cc:824] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:00.955992 22567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8: No bootstrap required, opened a new log
I20260926 19:44:00.963346 22567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8598a57db42646fcb57a566cb249e8b8" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43795 } }
I20260926 19:44:00.963657 22567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:00.963725 22567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8598a57db42646fcb57a566cb249e8b8, State: Initialized, Role: FOLLOWER
I20260926 19:44:00.964208 22567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8598a57db42646fcb57a566cb249e8b8" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43795 } }
I20260926 19:44:00.964411 22567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:00.964509 22567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:00.964627 22567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:00.966629 22567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8598a57db42646fcb57a566cb249e8b8" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43795 } }
I20260926 19:44:00.967051 22567 leader_election.cc:302] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8598a57db42646fcb57a566cb249e8b8; no voters:
I20260926 19:44:00.967561 22567 leader_election.cc:288] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:00.967681 22572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:00.968994 22572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [term 1 LEADER]: Becoming Leader. State: Replica: 8598a57db42646fcb57a566cb249e8b8, State: Running, Role: LEADER
I20260926 19:44:00.969467 22572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8598a57db42646fcb57a566cb249e8b8" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43795 } }
I20260926 19:44:00.969753 22567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:00.972209 22573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8598a57db42646fcb57a566cb249e8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8598a57db42646fcb57a566cb249e8b8" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43795 } } }
I20260926 19:44:00.972609 22573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:00.973418 22574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8598a57db42646fcb57a566cb249e8b8. Latest consensus state: current_term: 1 leader_uuid: "8598a57db42646fcb57a566cb249e8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8598a57db42646fcb57a566cb249e8b8" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 43795 } } }
I20260926 19:44:00.973769 22574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:00.979961 22581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:00.984251 22581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:00.992064 22581 catalog_manager.cc:1347] Generated new cluster ID: 0adb6ae31a0b4d3ebf0ea3aba822ae31
I20260926 19:44:00.992167 22581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:01.008085 22581 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:01.009271 22581 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:01.016894 22581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8: Generated new TSK 0
I20260926 19:44:01.017668 22581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:01.030671 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.65:0
--local_ip_for_outbound_sockets=127.2.164.65
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:43795
--builtin_ntp_servers=127.2.164.84:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:44:01.343226 22592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:01.343503 22592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:01.343629 22592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:01.350615 22592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:01.350715 22592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:01.350860 22592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.65
I20260926 19:44:01.359480 22592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.164.65
--webserver_port=0
--tserver_master_addrs=127.2.164.126:43795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.65
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:01.360621 22592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:01.362241 22592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:01.374707 22597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:01.374730 22598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:01.375277 22592 server_base.cc:1034] running on GCE node
W20260926 19:44:01.375360 22600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:01.375967 22592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:01.376852 22592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:01.378052 22592 hybrid_clock.cc:648] HybridClock initialized: now 1790451841378006 us; error 40 us; skew 500 ppm
I20260926 19:44:01.378350 22592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:44:01.380465 22592 webserver.cc:460] Webserver started at http://127.2.164.65:39903/ using document root <none> and password file <none>
I20260926 19:44:01.381089 22592 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:01.381202 22592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:01.381489 22592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:01.384213 22592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/data/instance:
uuid: "7c0bb741a1384c558ba56b2a37860dad"
format_stamp: "Formatted at 2026-09-26 19:44:01 on dist-test-slave-mrw4"
server_key: "0717082bd6927601f0990e485230cbce"
server_key_iv: "8d5be0b9076324f556d073f39aaaeb99"
server_key_version: "encryptionkey@0"
I20260926 19:44:01.385016 22592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance:
uuid: "7c0bb741a1384c558ba56b2a37860dad"
format_stamp: "Formatted at 2026-09-26 19:44:01 on dist-test-slave-mrw4"
server_key: "0717082bd6927601f0990e485230cbce"
server_key_iv: "8d5be0b9076324f556d073f39aaaeb99"
server_key_version: "encryptionkey@0"
I20260926 19:44:01.389765 22592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 19:44:01.392967 22606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:01.394263 22592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:44:01.394410 22592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/wal
uuid: "7c0bb741a1384c558ba56b2a37860dad"
format_stamp: "Formatted at 2026-09-26 19:44:01 on dist-test-slave-mrw4"
server_key: "0717082bd6927601f0990e485230cbce"
server_key_iv: "8d5be0b9076324f556d073f39aaaeb99"
server_key_version: "encryptionkey@0"
I20260926 19:44:01.394624 22592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:01.417337 22592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:01.418093 22592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:01.418358 22592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:01.419220 22592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:44:01.420593 22592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:44:01.420683 22592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:01.420783 22592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:44:01.420848 22592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:01.448907 22592 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.65:35225
I20260926 19:44:01.448946 22718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.65:35225 every 8 connection(s)
I20260926 19:44:01.450181 22592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/data/info.pb
I20260926 19:44:01.456638 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 22592
I20260926 19:44:01.456792 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/wal/instance
I20260926 19:44:01.457333 2705 external_mini_cluster.cc:1428] Setting key 2d3d2201fcb85c2bdab32462781ae1e4
I20260926 19:44:01.459900 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.66:0
--local_ip_for_outbound_sockets=127.2.164.66
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:43795
--builtin_ntp_servers=127.2.164.84:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:44:01.464797 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:01.453099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.65:60919 (local address 127.2.164.126:43795)
0926 19:44:01.453664 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:01.453671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:01.457233 (+ 3562us) server_negotiation.cc:407] Connection header received
0926 19:44:01.458031 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:01.458038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:01.458128 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:01.458228 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:01.459365 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:01.460143 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:01.462267 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:01.462703 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:01.462796 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:01.464158 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:01.464162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:01.464165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:01.464182 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:01.464220 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:01.464231 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:01.464291 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:01.464497 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 19:44:01.464569 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":274,"threads_started":1}
I20260926 19:44:01.466068 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:01.453830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:43795 (local address 127.2.164.65:60919)
0926 19:44:01.455422 (+ 1592us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:01.455471 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:01.457735 (+ 2264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:01.458474 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:01.458483 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:01.458581 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:01.459185 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:01.459208 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:01.460292 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:01.460297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:01.462136 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:01.462150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:01.462906 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:01.462912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:01.463181 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:01.463824 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:44:01.463828 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:01.463873 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:44:01.463967 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:01.464308 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:01.464316 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:01.464323 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 19:44:01.464477 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 19:44:01.464841 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":853,"threads_started":1}
I20260926 19:44:01.467464 22719 heartbeater.cc:347] Connected to a master server at 127.2.164.126:43795
I20260926 19:44:01.467756 22719 heartbeater.cc:464] Registering TS with master...
I20260926 19:44:01.468545 22719 heartbeater.cc:511] Master 127.2.164.126:43795 requested a full tablet report, sending...
I20260926 19:44:01.470675 22532 ts_manager.cc:194] Registered new tserver with Master: 7c0bb741a1384c558ba56b2a37860dad (127.2.164.65:35225)
I20260926 19:44:01.472600 22532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.65:60919
W20260926 19:44:01.767912 22723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:01.768160 22723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:01.768254 22723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:01.775355 22723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:01.775453 22723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:01.775564 22723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.66
I20260926 19:44:01.784180 22723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.164.66
--webserver_port=0
--tserver_master_addrs=127.2.164.126:43795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.66
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:01.785343 22723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:01.787016 22723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:01.799038 22731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:01.799072 22729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:01.799396 22728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:01.799750 22723 server_base.cc:1034] running on GCE node
I20260926 19:44:01.800282 22723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:01.801110 22723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:01.802325 22723 hybrid_clock.cc:648] HybridClock initialized: now 1790451841802267 us; error 47 us; skew 500 ppm
I20260926 19:44:01.802611 22723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:44:01.804642 22723 webserver.cc:460] Webserver started at http://127.2.164.66:42537/ using document root <none> and password file <none>
I20260926 19:44:01.805294 22723 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:01.805392 22723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:01.805677 22723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:01.808284 22723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/data/instance:
uuid: "d587e323f70f4b66a1c89f0a87bc2eaf"
format_stamp: "Formatted at 2026-09-26 19:44:01 on dist-test-slave-mrw4"
server_key: "93e78d342922ed59df256116098e41df"
server_key_iv: "c1dfffad179fb96b654d6067577e57d4"
server_key_version: "encryptionkey@0"
I20260926 19:44:01.809010 22723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance:
uuid: "d587e323f70f4b66a1c89f0a87bc2eaf"
format_stamp: "Formatted at 2026-09-26 19:44:01 on dist-test-slave-mrw4"
server_key: "93e78d342922ed59df256116098e41df"
server_key_iv: "c1dfffad179fb96b654d6067577e57d4"
server_key_version: "encryptionkey@0"
I20260926 19:44:01.813621 22723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:44:01.816747 22737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:01.818003 22723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:44:01.818159 22723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/wal
uuid: "d587e323f70f4b66a1c89f0a87bc2eaf"
format_stamp: "Formatted at 2026-09-26 19:44:01 on dist-test-slave-mrw4"
server_key: "93e78d342922ed59df256116098e41df"
server_key_iv: "c1dfffad179fb96b654d6067577e57d4"
server_key_version: "encryptionkey@0"
I20260926 19:44:01.818351 22723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:01.830339 22723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:01.831044 22723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:01.831310 22723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:01.832166 22723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:44:01.833580 22723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:44:01.833674 22723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:01.833775 22723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:44:01.833842 22723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:01.862746 22723 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.66:44535
I20260926 19:44:01.862804 22849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.66:44535 every 8 connection(s)
I20260926 19:44:01.863965 22723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/data/info.pb
I20260926 19:44:01.874222 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 22723
I20260926 19:44:01.874372 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/wal/instance
I20260926 19:44:01.874815 2705 external_mini_cluster.cc:1428] Setting key b9cda71e0308c773f50f4b3c23a46bf5
I20260926 19:44:01.875674 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:01.866389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.66:54531 (local address 127.2.164.126:43795)
0926 19:44:01.866573 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:01.866578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:01.867890 (+ 1312us) server_negotiation.cc:407] Connection header received
0926 19:44:01.869658 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:01.869663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:01.869732 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:01.869823 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:01.870876 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:01.871639 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:01.873215 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:01.873636 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:01.873724 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:01.874963 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:01.874970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:01.874974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:01.874995 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:01.875033 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:01.875049 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:01.875138 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:01.875311 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 19:44:01.875394 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:44:01.876678 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:01.866857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:43795 (local address 127.2.164.66:54531)
0926 19:44:01.867725 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:01.867763 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:01.869357 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:01.870046 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:01.870056 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:01.870144 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:01.870736 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:01.870754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:01.871772 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:01.871776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:01.873077 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:01.873089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:01.873805 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:01.873810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:01.874010 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:01.874672 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:44:01.874674 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:01.874710 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:44:01.874781 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:01.875134 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:01.875141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:01.875146 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:44:01.875275 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 19:44:01.875617 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":267,"threads_started":1}
I20260926 19:44:01.877571 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.164.67:0
--local_ip_for_outbound_sockets=127.2.164.67
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:43795
--builtin_ntp_servers=127.2.164.84:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:44:01.877936 22850 heartbeater.cc:347] Connected to a master server at 127.2.164.126:43795
I20260926 19:44:01.878196 22850 heartbeater.cc:464] Registering TS with master...
I20260926 19:44:01.878934 22850 heartbeater.cc:511] Master 127.2.164.126:43795 requested a full tablet report, sending...
I20260926 19:44:01.880184 22532 ts_manager.cc:194] Registered new tserver with Master: d587e323f70f4b66a1c89f0a87bc2eaf (127.2.164.66:44535)
I20260926 19:44:01.880962 22532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.66:54531
W20260926 19:44:02.180256 22853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:02.180537 22853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:02.180670 22853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:02.187645 22853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:02.187739 22853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:02.187878 22853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.164.67
I20260926 19:44:02.196514 22853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.164.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.164.67
--webserver_port=0
--tserver_master_addrs=127.2.164.126:43795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.164.67
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:02.197721 22853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:02.199510 22853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:02.211426 22858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:02.211457 22861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:02.211479 22859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:02.212231 22853 server_base.cc:1034] running on GCE node
I20260926 19:44:02.212762 22853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:02.213649 22853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:02.214845 22853 hybrid_clock.cc:648] HybridClock initialized: now 1790451842214805 us; error 30 us; skew 500 ppm
I20260926 19:44:02.215113 22853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:44:02.217207 22853 webserver.cc:460] Webserver started at http://127.2.164.67:40119/ using document root <none> and password file <none>
I20260926 19:44:02.217828 22853 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:02.217909 22853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:02.218161 22853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:02.220774 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/data/instance:
uuid: "37eb7fa13c494d9f84f4e3555bade5ba"
format_stamp: "Formatted at 2026-09-26 19:44:02 on dist-test-slave-mrw4"
server_key: "7d07358960f0753a2637005158d422f9"
server_key_iv: "ae9d876d7368ada15b2a0a95f333bd06"
server_key_version: "encryptionkey@0"
I20260926 19:44:02.221558 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance:
uuid: "37eb7fa13c494d9f84f4e3555bade5ba"
format_stamp: "Formatted at 2026-09-26 19:44:02 on dist-test-slave-mrw4"
server_key: "7d07358960f0753a2637005158d422f9"
server_key_iv: "ae9d876d7368ada15b2a0a95f333bd06"
server_key_version: "encryptionkey@0"
I20260926 19:44:02.226199 22853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:44:02.229353 22867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:02.230562 22853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:44:02.230679 22853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/wal
uuid: "37eb7fa13c494d9f84f4e3555bade5ba"
format_stamp: "Formatted at 2026-09-26 19:44:02 on dist-test-slave-mrw4"
server_key: "7d07358960f0753a2637005158d422f9"
server_key_iv: "ae9d876d7368ada15b2a0a95f333bd06"
server_key_version: "encryptionkey@0"
I20260926 19:44:02.230841 22853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:02.247910 22853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:02.248615 22853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:02.248873 22853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:02.249778 22853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:44:02.251070 22853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:44:02.251156 22853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:02.251252 22853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:44:02.251317 22853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:02.278031 22853 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.67:45961
I20260926 19:44:02.278077 22979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.67:45961 every 8 connection(s)
I20260926 19:44:02.279294 22853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/data/info.pb
I20260926 19:44:02.288064 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 22853
I20260926 19:44:02.288197 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/wal/instance
I20260926 19:44:02.288721 2705 external_mini_cluster.cc:1428] Setting key 572d1fa34ada5f100c1d2a7b72fe08d3
I20260926 19:44:02.291267 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.281702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:47689 (local address 127.2.164.126:43795)
0926 19:44:02.281915 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:02.281921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:02.283255 (+ 1334us) server_negotiation.cc:407] Connection header received
0926 19:44:02.284889 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:02.284894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:02.284959 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:02.285045 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:02.286037 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.286798 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.288467 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.289041 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.289196 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.290543 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:02.290549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:02.290554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:02.290585 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:02.290624 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:02.290638 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:02.290711 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:02.290924 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 19:44:02.290997 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:44:02.292378 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.282282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:43795 (local address 127.2.164.67:47689)
0926 19:44:02.283113 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:02.283153 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:02.284655 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:02.285266 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:02.285274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:02.285357 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:02.285879 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.285895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.286923 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.286926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.288323 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.288335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.289267 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.289274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.289507 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.290208 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:44:02.290211 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:02.290260 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:44:02.290359 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:02.290731 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:02.290741 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:02.290749 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 19:44:02.290920 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 19:44:02.291296 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":248,"threads_started":1}
I20260926 19:44:02.293568 22980 heartbeater.cc:347] Connected to a master server at 127.2.164.126:43795
I20260926 19:44:02.293812 22980 heartbeater.cc:464] Registering TS with master...
I20260926 19:44:02.294479 22980 heartbeater.cc:511] Master 127.2.164.126:43795 requested a full tablet report, sending...
I20260926 19:44:02.295712 22532 ts_manager.cc:194] Registered new tserver with Master: 37eb7fa13c494d9f84f4e3555bade5ba (127.2.164.67:45961)
I20260926 19:44:02.296445 22532 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.164.67:47689
I20260926 19:44:02.303588 2705 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:44:02.312264 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.307340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40584 (local address 127.2.164.126:43795)
0926 19:44:02.307488 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:02.307492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:02.307966 (+ 474us) server_negotiation.cc:407] Connection header received
0926 19:44:02.308094 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:02.308098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:02.308165 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:02.308241 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:02.308624 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.309324 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.310498 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.310862 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.310946 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.311788 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:02.311793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:02.311796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:02.311810 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:02.311836 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:02.311846 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:02.311905 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:02.311982 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 19:44:02.312044 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 19:44:02.315519 22532 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:44:02.321954 22532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:44:02.341724 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.332569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.67:45961 (local address 127.0.0.1:44586)
0926 19:44:02.333298 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:02.333387 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:02.333569 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:02.334469 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:02.334475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:02.334504 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:44:02.334902 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.334914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.337319 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.337325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.339404 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.339417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.340858 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.340864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.341054 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.341103 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 19:44:02.341244 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 19:44:02.341406 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":310,"threads_started":1}
I20260926 19:44:02.341797 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.332754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.2.164.67:45961)
0926 19:44:02.333866 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:02.333874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:02.333905 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:44:02.334009 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:02.334017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:02.334260 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:02.335111 (+ 851us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:44:02.335371 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.337119 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.339630 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.340683 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.340807 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.341077 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:02.341286 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 19:44:02.341448 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":792,"threads_started":1}
I20260926 19:44:02.342870 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.333689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35225 (local address 127.0.0.1:47308)
0926 19:44:02.334177 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:02.334192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:02.334289 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:02.336049 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:02.336054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:02.336080 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:44:02.336468 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.336479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.338088 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.338097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.340258 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.340272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.342060 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.342066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.342250 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.342283 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:44:02.342344 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 19:44:02.342487 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":71,"thread_start_us":210,"threads_started":1}
I20260926 19:44:02.342890 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.334661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.2.164.66:44535)
0926 19:44:02.335747 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:02.335755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:02.335780 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:44:02.335869 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:02.335875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:02.336086 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:02.336798 (+ 712us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:44:02.336972 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.338370 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.340672 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.341983 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.342096 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.342345 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:02.342425 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 19:44:02.342579 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":820,"threads_started":1}
I20260926 19:44:02.342890 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.334215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:44535 (local address 127.0.0.1:48930)
0926 19:44:02.334862 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:02.334881 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:02.334980 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:02.336287 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:02.336295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:02.336320 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:44:02.336645 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.336654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.338510 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.338515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.340522 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.340532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.342141 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.342146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.342302 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.342328 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 19:44:02.342388 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 19:44:02.342514 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":26,"thread_start_us":578,"threads_started":1}
I20260926 19:44:02.344331 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.334115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.2.164.65:35225)
0926 19:44:02.335489 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:02.335504 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:02.335530 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:44:02.335618 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:02.335625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:02.335846 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:02.336653 (+ 807us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:44:02.336866 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.337969 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.340452 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.341859 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.343560 (+ 1701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.343780 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:02.343841 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 19:44:02.344020 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"thread_start_us":1052,"threads_started":1}
I20260926 19:44:02.345817 22915 tablet_service.cc:1467] Processing CreateTablet for tablet 3d64ded581cc45ad9395eb35fc86f220 (DEFAULT_TABLE table=test-table [id=41da1d8aeeff4922a18a4770afb6fe0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:44:02.346170 22785 tablet_service.cc:1467] Processing CreateTablet for tablet 3d64ded581cc45ad9395eb35fc86f220 (DEFAULT_TABLE table=test-table [id=41da1d8aeeff4922a18a4770afb6fe0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:44:02.346994 22915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d64ded581cc45ad9395eb35fc86f220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:02.347474 22785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d64ded581cc45ad9395eb35fc86f220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:02.347500 22654 tablet_service.cc:1467] Processing CreateTablet for tablet 3d64ded581cc45ad9395eb35fc86f220 (DEFAULT_TABLE table=test-table [id=41da1d8aeeff4922a18a4770afb6fe0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:44:02.348798 22654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d64ded581cc45ad9395eb35fc86f220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:02.357312 22998 tablet_bootstrap.cc:492] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: Bootstrap starting.
I20260926 19:44:02.360107 22998 tablet_bootstrap.cc:654] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:02.361238 22998 log.cc:824] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:02.362159 23000 tablet_bootstrap.cc:492] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: Bootstrap starting.
I20260926 19:44:02.365626 23000 tablet_bootstrap.cc:654] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:02.365859 22999 tablet_bootstrap.cc:492] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: Bootstrap starting.
I20260926 19:44:02.366715 23000 log.cc:824] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:02.368413 22998 tablet_bootstrap.cc:492] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: No bootstrap required, opened a new log
I20260926 19:44:02.368666 22998 ts_tablet_manager.cc:1397] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260926 19:44:02.369235 22999 tablet_bootstrap.cc:654] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:02.369532 23000 tablet_bootstrap.cc:492] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: No bootstrap required, opened a new log
I20260926 19:44:02.369823 23000 ts_tablet_manager.cc:1397] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 19:44:02.370455 22999 log.cc:824] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:02.372725 22999 tablet_bootstrap.cc:492] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: No bootstrap required, opened a new log
I20260926 19:44:02.372934 22999 ts_tablet_manager.cc:1397] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 19:44:02.378703 22998 raft_consensus.cc:348] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45961 } } peers { permanent_uuid: "7c0bb741a1384c558ba56b2a37860dad" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35225 } } peers { permanent_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44535 } }
I20260926 19:44:02.379158 22998 raft_consensus.cc:374] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:02.379258 22998 raft_consensus.cc:729] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37eb7fa13c494d9f84f4e3555bade5ba, State: Initialized, Role: FOLLOWER
I20260926 19:44:02.379657 23000 raft_consensus.cc:348] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45961 } } peers { permanent_uuid: "7c0bb741a1384c558ba56b2a37860dad" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35225 } } peers { permanent_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44535 } }
I20260926 19:44:02.379832 22998 consensus_queue.cc:260] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45961 } } peers { permanent_uuid: "7c0bb741a1384c558ba56b2a37860dad" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35225 } } peers { permanent_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44535 } }
I20260926 19:44:02.380108 23000 raft_consensus.cc:374] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:02.379858 22999 raft_consensus.cc:348] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45961 } } peers { permanent_uuid: "7c0bb741a1384c558ba56b2a37860dad" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35225 } } peers { permanent_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44535 } }
I20260926 19:44:02.380230 23000 raft_consensus.cc:729] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0bb741a1384c558ba56b2a37860dad, State: Initialized, Role: FOLLOWER
I20260926 19:44:02.380333 22999 raft_consensus.cc:374] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:02.380479 22999 raft_consensus.cc:729] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d587e323f70f4b66a1c89f0a87bc2eaf, State: Initialized, Role: FOLLOWER
I20260926 19:44:02.380895 23000 consensus_queue.cc:260] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45961 } } peers { permanent_uuid: "7c0bb741a1384c558ba56b2a37860dad" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35225 } } peers { permanent_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44535 } }
I20260926 19:44:02.381028 22999 consensus_queue.cc:260] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45961 } } peers { permanent_uuid: "7c0bb741a1384c558ba56b2a37860dad" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35225 } } peers { permanent_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44535 } }
I20260926 19:44:02.382011 22850 heartbeater.cc:503] Master 127.2.164.126:43795 was elected leader, sending a full tablet report...
I20260926 19:44:02.382694 22999 ts_tablet_manager.cc:1428] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260926 19:44:02.384951 22719 heartbeater.cc:503] Master 127.2.164.126:43795 was elected leader, sending a full tablet report...
I20260926 19:44:02.385882 23000 ts_tablet_manager.cc:1428] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260926 19:44:02.386662 22980 heartbeater.cc:503] Master 127.2.164.126:43795 was elected leader, sending a full tablet report...
I20260926 19:44:02.387226 22998 ts_tablet_manager.cc:1428] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: Time spent starting tablet: real 0.018s user 0.009s sys 0.010s
W20260926 19:44:02.452632 22720 tablet.cc:2369] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:44:02.530897 22981 tablet.cc:2369] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:44:02.541544 23004 raft_consensus.cc:482] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:44:02.541785 23004 raft_consensus.cc:504] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45961 } } peers { permanent_uuid: "7c0bb741a1384c558ba56b2a37860dad" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35225 } } peers { permanent_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44535 } }
I20260926 19:44:02.543323 23004 leader_election.cc:288] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0bb741a1384c558ba56b2a37860dad (127.2.164.65:35225), d587e323f70f4b66a1c89f0a87bc2eaf (127.2.164.66:44535)
I20260926 19:44:02.549503 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.543601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.65:35225 (local address 127.2.164.67:36053)
0926 19:44:02.543761 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:02.543777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:02.543879 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:02.544488 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:02.544493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:02.544521 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:44:02.544711 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.544721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.545876 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.545879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.547692 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.547711 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.548934 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.548938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.549065 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.549089 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 19:44:02.549152 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 19:44:02.549275 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 19:44:02.549592 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.543862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:36053 (local address 127.2.164.65:35225)
0926 19:44:02.544063 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:02.544069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:02.544090 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:44:02.544167 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:02.544174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:02.544266 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:02.544357 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:44:02.544861 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.545734 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.547845 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.548794 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.548931 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.548999 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:02.549214 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 19:44:02.549339 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 19:44:02.550134 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.543974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.66:44535 (local address 127.2.164.67:55803)
0926 19:44:02.544582 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:02.544597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:02.544715 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:02.545030 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:02.545033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:02.545053 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:44:02.545364 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.545373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.546414 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.546420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.548189 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:02.548200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.549577 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.549583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:02.549749 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.549778 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 19:44:02.549855 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 19:44:02.549938 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":290,"threads_started":1}
I20260926 19:44:02.550232 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.544174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.164.67:55803 (local address 127.2.164.66:44535)
0926 19:44:02.544358 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:02.544364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:02.544685 (+ 321us) server_negotiation.cc:407] Connection header received
0926 19:44:02.544837 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:02.544842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:02.544914 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:02.544998 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:44:02.545506 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.546275 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.548352 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.549408 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.549544 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.549616 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:02.549899 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 19:44:02.550017 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:44:02.550685 22674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64ded581cc45ad9395eb35fc86f220" candidate_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bb741a1384c558ba56b2a37860dad" is_pre_election: true
I20260926 19:44:02.551244 22674 raft_consensus.cc:2454] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37eb7fa13c494d9f84f4e3555bade5ba in term 0.
I20260926 19:44:02.551172 22805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64ded581cc45ad9395eb35fc86f220" candidate_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" is_pre_election: true
I20260926 19:44:02.551645 22805 raft_consensus.cc:2454] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37eb7fa13c494d9f84f4e3555bade5ba in term 0.
I20260926 19:44:02.551906 22871 leader_election.cc:302] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37eb7fa13c494d9f84f4e3555bade5ba, 7c0bb741a1384c558ba56b2a37860dad; no voters:
I20260926 19:44:02.552234 23004 raft_consensus.cc:2781] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:44:02.552356 23004 raft_consensus.cc:482] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:44:02.552441 23004 raft_consensus.cc:3037] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:02.554529 23004 raft_consensus.cc:504] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45961 } } peers { permanent_uuid: "7c0bb741a1384c558ba56b2a37860dad" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35225 } } peers { permanent_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44535 } }
I20260926 19:44:02.555029 23004 leader_election.cc:288] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [CANDIDATE]: Term 1 election: Requested vote from peers 7c0bb741a1384c558ba56b2a37860dad (127.2.164.65:35225), d587e323f70f4b66a1c89f0a87bc2eaf (127.2.164.66:44535)
I20260926 19:44:02.555441 22674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64ded581cc45ad9395eb35fc86f220" candidate_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0bb741a1384c558ba56b2a37860dad"
I20260926 19:44:02.555505 22805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64ded581cc45ad9395eb35fc86f220" candidate_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf"
I20260926 19:44:02.555610 22674 raft_consensus.cc:3037] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:02.555691 22805 raft_consensus.cc:3037] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:02.557776 22674 raft_consensus.cc:2454] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37eb7fa13c494d9f84f4e3555bade5ba in term 1.
I20260926 19:44:02.557821 22805 raft_consensus.cc:2454] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37eb7fa13c494d9f84f4e3555bade5ba in term 1.
I20260926 19:44:02.558122 22871 leader_election.cc:302] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37eb7fa13c494d9f84f4e3555bade5ba, 7c0bb741a1384c558ba56b2a37860dad; no voters:
I20260926 19:44:02.558364 23004 raft_consensus.cc:2781] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:02.558761 23004 raft_consensus.cc:686] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 1 LEADER]: Becoming Leader. State: Replica: 37eb7fa13c494d9f84f4e3555bade5ba, State: Running, Role: LEADER
I20260926 19:44:02.559191 23004 consensus_queue.cc:237] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45961 } } peers { permanent_uuid: "7c0bb741a1384c558ba56b2a37860dad" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35225 } } peers { permanent_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44535 } }
I20260926 19:44:02.563239 22531 catalog_manager.cc:5495] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 37eb7fa13c494d9f84f4e3555bade5ba (127.2.164.67). New cstate: current_term: 1 leader_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7fa13c494d9f84f4e3555bade5ba" member_type: VOTER last_known_addr { host: "127.2.164.67" port: 45961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c0bb741a1384c558ba56b2a37860dad" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44535 } health_report { overall_health: UNKNOWN } } }
W20260926 19:44:02.615821 22851 tablet.cc:2369] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:44:02.632768 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.628051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.2.164.67:45961)
0926 19:44:02.628209 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:02.628214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:02.628290 (+ 76us) server_negotiation.cc:407] Connection header received
0926 19:44:02.628493 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:02.628497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:02.628582 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:02.628695 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:44:02.628961 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.629685 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.630994 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.631373 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.631459 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.631874 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:44:02.632354 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:44:02.632465 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:02.632531 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 19:44:02.632586 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 19:44:02.640270 22805 raft_consensus.cc:1261] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf [term 1 FOLLOWER]: Refusing update from remote peer 37eb7fa13c494d9f84f4e3555bade5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:44:02.640343 22674 raft_consensus.cc:1261] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad [term 1 FOLLOWER]: Refusing update from remote peer 37eb7fa13c494d9f84f4e3555bade5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:44:02.641119 23008 consensus_queue.cc:1035] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d587e323f70f4b66a1c89f0a87bc2eaf" member_type: VOTER last_known_addr { host: "127.2.164.66" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:44:02.641479 23004 consensus_queue.cc:1035] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0bb741a1384c558ba56b2a37860dad" member_type: VOTER last_known_addr { host: "127.2.164.65" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:44:02.664067 22532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:44:02.664292 22532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:44:02.667666 22532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8598a57db42646fcb57a566cb249e8b8: Sending DeleteTablet for 3 replicas of tablet 3d64ded581cc45ad9395eb35fc86f220
I20260926 19:44:02.668850 22915 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d64ded581cc45ad9395eb35fc86f220 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:44:02 UTC) from {username='slave'} at 127.0.0.1:44586
I20260926 19:44:02.668881 22654 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d64ded581cc45ad9395eb35fc86f220 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:44:02 UTC) from {username='slave'} at 127.0.0.1:47308
I20260926 19:44:02.669508 22785 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d64ded581cc45ad9395eb35fc86f220 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:44:02 UTC) from {username='slave'} at 127.0.0.1:48930
I20260926 19:44:02.669644 23022 tablet_replica.cc:330] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: stopping tablet replica
I20260926 19:44:02.669708 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 22592
I20260926 19:44:02.670058 23022 raft_consensus.cc:2229] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:44:02.670756 23022 raft_consensus.cc:2258] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:44:02.671504 23021 tablet_replica.cc:330] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: stopping tablet replica
I20260926 19:44:02.671900 23021 raft_consensus.cc:2229] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:44:02.672386 23021 raft_consensus.cc:2258] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:44:02.674256 23022 ts_tablet_manager.cc:1905] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:44:02.677289 23021 ts_tablet_manager.cc:1905] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:44:02.681761 23022 ts_tablet_manager.cc:1918] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:44:02.681950 23022 log.cc:1196] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-2/wal/wals/3d64ded581cc45ad9395eb35fc86f220
I20260926 19:44:02.682514 23022 ts_tablet_manager.cc:1939] T 3d64ded581cc45ad9395eb35fc86f220 P 37eb7fa13c494d9f84f4e3555bade5ba: Deleting consensus metadata
I20260926 19:44:02.684062 22519 catalog_manager.cc:4841] TS 37eb7fa13c494d9f84f4e3555bade5ba (127.2.164.67:45961): tablet 3d64ded581cc45ad9395eb35fc86f220 (table test-table [id=41da1d8aeeff4922a18a4770afb6fe0d]) successfully deleted
I20260926 19:44:02.685639 23023 tablet_replica.cc:330] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: stopping tablet replica
I20260926 19:44:02.686009 23023 raft_consensus.cc:2229] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:44:02.686444 23023 raft_consensus.cc:2258] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:44:02.688409 23021 ts_tablet_manager.cc:1918] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:44:02.688620 23021 log.cc:1196] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-0/wal/wals/3d64ded581cc45ad9395eb35fc86f220
I20260926 19:44:02.688807 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.671047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.2.164.65:35225)
0926 19:44:02.671222 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:02.671228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:02.681069 (+ 9841us) server_negotiation.cc:407] Connection header received
0926 19:44:02.681475 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:02.681481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:02.681602 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:02.681679 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:02.682329 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.683345 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.685152 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.685684 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.685800 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.686885 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:02.686896 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:02.686911 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:02.687582 (+ 671us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:02.687645 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:02.687667 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:02.687786 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:02.688292 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 19:44:02.688419 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:44:02.689360 23021 ts_tablet_manager.cc:1939] T 3d64ded581cc45ad9395eb35fc86f220 P 7c0bb741a1384c558ba56b2a37860dad: Deleting consensus metadata
I20260926 19:44:02.689477 23023 ts_tablet_manager.cc:1905] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:44:02.689518 22714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:02.696527 23023 ts_tablet_manager.cc:1918] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:44:02.696671 23023 log.cc:1196] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: Deleting WAL directory at /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790451540687502-2705-0/minicluster-data/ts-1/wal/wals/3d64ded581cc45ad9395eb35fc86f220
I20260926 19:44:02.697152 23023 ts_tablet_manager.cc:1939] T 3d64ded581cc45ad9395eb35fc86f220 P d587e323f70f4b66a1c89f0a87bc2eaf: Deleting consensus metadata
I20260926 19:44:02.698771 22519 catalog_manager.cc:4841] TS d587e323f70f4b66a1c89f0a87bc2eaf (127.2.164.66:44535): tablet 3d64ded581cc45ad9395eb35fc86f220 (table test-table [id=41da1d8aeeff4922a18a4770afb6fe0d]) successfully deleted
I20260926 19:44:02.779557 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 22592
I20260926 19:44:02.779809 22519 catalog_manager.cc:4841] TS 7c0bb741a1384c558ba56b2a37860dad (127.2.164.65:35225): tablet 3d64ded581cc45ad9395eb35fc86f220 (table test-table [id=41da1d8aeeff4922a18a4770afb6fe0d]) successfully deleted
I20260926 19:44:02.793934 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 22723
I20260926 19:44:02.801226 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.794763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.2.164.66:44535)
0926 19:44:02.794939 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:02.794944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:02.794965 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:44:02.795058 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:02.795064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:02.795197 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:02.795290 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:02.795709 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.796444 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.798260 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.798690 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.798789 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.799914 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:02.799923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:02.799938 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:02.800504 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:02.800573 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:02.800593 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:02.800676 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:02.800811 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 19:44:02.800896 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:44:02.801841 22845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:02.890125 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 22723
I20260926 19:44:02.904625 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 22853
I20260926 19:44:02.912320 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:02.905381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.2.164.67:45961)
0926 19:44:02.905625 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:02.905631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:02.905660 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:44:02.905728 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:02.905733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:02.905839 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:02.905976 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:02.906322 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.907531 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.908804 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:02.909492 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:02.909623 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:02.910583 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:02.910596 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:02.910614 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:02.911439 (+ 825us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:02.911518 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:02.911544 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:02.911647 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:02.911809 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 19:44:02.911911 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 19:44:02.913064 22975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:03.029506 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 22853
I20260926 19:44:03.044919 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 22501
I20260926 19:44:03.046003 22562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:03.152514 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 22501
2026-09-26T19:44:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2702 ms)
[----------] 6 tests from EncryptionPolicyTest (19652 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:03 dist-test-slave-mrw4 krb5kdc[23031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:03 dist-test-slave-mrw4 krb5kdc[23031](info): set up 2 sockets
Sep 26 19:44:03 dist-test-slave-mrw4 krb5kdc[23031](info): commencing operation
krb5kdc: starting...
W20260926 19:44:04.161423 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:04 dist-test-slave-mrw4 krb5kdc[23031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:04.388036 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:04.695582 23044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:04.695868 23044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:04.696005 23044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:04.703001 23044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:04.703109 23044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:04.703184 23044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:04.703263 23044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:04.703322 23044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:04.712811 23044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:04.714022 23044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:04.715233 23044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:04.727023 23049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:04.727064 23050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:04.727093 23052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:04.728111 23044 server_base.cc:1034] running on GCE node
I20260926 19:44:04.728776 23044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:04.730060 23044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:04.731249 23044 hybrid_clock.cc:648] HybridClock initialized: now 1790451844731211 us; error 26 us; skew 500 ppm
I20260926 19:44:04.731539 23044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:04 dist-test-slave-mrw4 krb5kdc[23031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:04.739001 23044 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:04.740442 23044 webserver.cc:460] Webserver started at http://127.0.0.1:42211/ using document root <none> and password file <none>
I20260926 19:44:04.741087 23044 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:04.741221 23044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:04.741500 23044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:04.744264 23044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "4956d7511c9c4797a7b255f49da148f7"
format_stamp: "Formatted at 2026-09-26 19:44:04 on dist-test-slave-mrw4"
server_key: "dfdfbe6efd6e95482744864dedd40a98"
server_key_iv: "fa80ecb4bf7ee3970633884336609c90"
server_key_version: "encryptionkey@0"
I20260926 19:44:04.744978 23044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "4956d7511c9c4797a7b255f49da148f7"
format_stamp: "Formatted at 2026-09-26 19:44:04 on dist-test-slave-mrw4"
server_key: "dfdfbe6efd6e95482744864dedd40a98"
server_key_iv: "fa80ecb4bf7ee3970633884336609c90"
server_key_version: "encryptionkey@0"
I20260926 19:44:04.749738 23044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 19:44:04.752830 23059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:04.754303 23044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:44:04.754480 23044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "4956d7511c9c4797a7b255f49da148f7"
format_stamp: "Formatted at 2026-09-26 19:44:04 on dist-test-slave-mrw4"
server_key: "dfdfbe6efd6e95482744864dedd40a98"
server_key_iv: "fa80ecb4bf7ee3970633884336609c90"
server_key_version: "encryptionkey@0"
I20260926 19:44:04.754716 23044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:04.780237 23044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:04.784065 23044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:04.784307 23044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:04.799906 23044 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38601
I20260926 19:44:04.799937 23110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38601 every 8 connection(s)
I20260926 19:44:04.801129 23044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:04.804816 23111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:04.808394 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 23044
I20260926 19:44:04.808601 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:04.809423 2705 external_mini_cluster.cc:1428] Setting key f5f59444d744bf620d6eac67c7fe20b2
I20260926 19:44:04.817286 23111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7: Bootstrap starting.
I20260926 19:44:04.820703 23111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:04.821936 23111 log.cc:824] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:44:04 dist-test-slave-mrw4 krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 19:44:04.824808 23111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7: No bootstrap required, opened a new log
I20260926 19:44:04.831228 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:04.811178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41825 (local address 127.0.0.1:38601)
0926 19:44:04.811879 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:04.811890 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:04.811930 (+ 40us) server_negotiation.cc:407] Connection header received
0926 19:44:04.813486 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:04.813506 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:04.813886 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:04.814205 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:04.815826 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:04.817089 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:04.818797 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:04.819440 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:04.819556 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:04.824350 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:04.824357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:04.824370 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:04.824403 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:04.827287 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:04.828162 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:04.828167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:04.828172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:04.828331 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:04.828769 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:04.828776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:04.828778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:04.829314 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:04.829530 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:04.829983 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 19:44:04.830259 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":342,"threads_started":1}
I20260926 19:44:04.833316 23111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4956d7511c9c4797a7b255f49da148f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260926 19:44:04.833725 23111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:04.833822 23111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4956d7511c9c4797a7b255f49da148f7, State: Initialized, Role: FOLLOWER
I20260926 19:44:04.834410 23111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4956d7511c9c4797a7b255f49da148f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260926 19:44:04.834654 23111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:04.834774 23111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:04.834921 23111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:04.835125 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:04.836922 23111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4956d7511c9c4797a7b255f49da148f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260926 19:44:04.837383 23111 leader_election.cc:302] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4956d7511c9c4797a7b255f49da148f7; no voters:
I20260926 19:44:04.837949 23111 leader_election.cc:288] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:04.838117 23117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:04.839227 23117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [term 1 LEADER]: Becoming Leader. State: Replica: 4956d7511c9c4797a7b255f49da148f7, State: Running, Role: LEADER
I20260926 19:44:04.839742 23117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4956d7511c9c4797a7b255f49da148f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260926 19:44:04.840216 23111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:04.842370 23121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4956d7511c9c4797a7b255f49da148f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4956d7511c9c4797a7b255f49da148f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } } }
I20260926 19:44:04.842751 23121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:04.843230 23122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4956d7511c9c4797a7b255f49da148f7. Latest consensus state: current_term: 1 leader_uuid: "4956d7511c9c4797a7b255f49da148f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4956d7511c9c4797a7b255f49da148f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } } }
I20260926 19:44:04.843532 23122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:04.857062 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:04.839937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47705 (local address 127.0.0.1:38601)
0926 19:44:04.840120 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:04.840126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:04.840374 (+ 248us) server_negotiation.cc:407] Connection header received
0926 19:44:04.840547 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:04.840553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:04.840623 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:04.840701 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:04.842427 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:04.843610 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:04.845638 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:04.846322 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:04.846420 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:04.849752 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:04.849760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:04.849765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:04.849804 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:04.853449 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:04.854395 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:04.854401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:04.854405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:04.854549 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:04.855072 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:04.855077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:04.855081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:04.855516 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:04.855777 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:04.856495 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 19:44:04.856686 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:44:04.867401 23126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:04.871531 23126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:04.880373 23126 catalog_manager.cc:1347] Generated new cluster ID: 6a2f6aedca8846debbdbbc604c819743
I20260926 19:44:04.880475 23126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:04.904040 23126 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:04.905321 23126 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:04.915642 23126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4956d7511c9c4797a7b255f49da148f7: Generated new TSK 0
I20260926 19:44:04.916442 23126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:04.929996 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23044
I20260926 19:44:04.931053 23106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:05.021600 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 23044
W20260926 19:44:05.035710 23029 connection.cc:535] client connection to 127.0.0.1:38601 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38601): Connection reset by peer (error 104)
2026-09-26T19:44:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1865 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:05 dist-test-slave-mrw4 krb5kdc[23144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:05 dist-test-slave-mrw4 krb5kdc[23144](info): set up 2 sockets
Sep 26 19:44:05 dist-test-slave-mrw4 krb5kdc[23144](info): commencing operation
krb5kdc: starting...
W20260926 19:44:06.021294 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:06 dist-test-slave-mrw4 krb5kdc[23144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:06.241542 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:06.567534 23157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:06.567929 23157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:06.568095 23157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:06.575099 23157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:06.575219 23157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:06.575299 23157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:06.575380 23157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:06.575439 23157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:06.584852 23157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37139
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:06.586078 23157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:06.587323 23157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:06.597990 23163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:06.598199 23162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:06.598497 23165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:06.598889 23157 server_base.cc:1034] running on GCE node
I20260926 19:44:06.599571 23157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:06.600816 23157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:06.601997 23157 hybrid_clock.cc:648] HybridClock initialized: now 1790451846601949 us; error 38 us; skew 500 ppm
I20260926 19:44:06.602311 23157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:06 dist-test-slave-mrw4 krb5kdc[23144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:06.605123 23157 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:06.606534 23157 webserver.cc:460] Webserver started at http://127.0.0.1:36763/ using document root <none> and password file <none>
I20260926 19:44:06.607146 23157 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:06.607254 23157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:06.607509 23157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:06.610241 23157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "5ff9aefb6dc74deeb2d1fed66f80b059"
format_stamp: "Formatted at 2026-09-26 19:44:06 on dist-test-slave-mrw4"
server_key: "ea38072bb68195c95e690e03098fa2a2"
server_key_iv: "9246e088cfa5ee24c43ca185634a83a3"
server_key_version: "encryptionkey@0"
I20260926 19:44:06.610924 23157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "5ff9aefb6dc74deeb2d1fed66f80b059"
format_stamp: "Formatted at 2026-09-26 19:44:06 on dist-test-slave-mrw4"
server_key: "ea38072bb68195c95e690e03098fa2a2"
server_key_iv: "9246e088cfa5ee24c43ca185634a83a3"
server_key_version: "encryptionkey@0"
I20260926 19:44:06.615543 23157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:44:06.618643 23172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:06.619814 23157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 19:44:06.619961 23157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "5ff9aefb6dc74deeb2d1fed66f80b059"
format_stamp: "Formatted at 2026-09-26 19:44:06 on dist-test-slave-mrw4"
server_key: "ea38072bb68195c95e690e03098fa2a2"
server_key_iv: "9246e088cfa5ee24c43ca185634a83a3"
server_key_version: "encryptionkey@0"
I20260926 19:44:06.620126 23157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:06.647661 23157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:06.651326 23157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:06.651561 23157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:06.667997 23157 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37139
I20260926 19:44:06.668051 23223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37139 every 8 connection(s)
I20260926 19:44:06.669241 23157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:06.671924 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 23157
I20260926 19:44:06.672108 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:06.672621 2705 external_mini_cluster.cc:1428] Setting key c0122d019cabbfe37443242923a58888
I20260926 19:44:06.673025 23224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:06.685959 23224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059: Bootstrap starting.
I20260926 19:44:06.686014 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:06.674264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57639 (local address 127.0.0.1:37139)
0926 19:44:06.674964 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:06.674975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:06.675012 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:44:06.677461 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:06.677482 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:06.677865 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:06.678219 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:06.678783 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:06.680107 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:06.681844 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:06.682505 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:06.682636 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:06.683835 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:06.683842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:06.683854 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:06.684373 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:06.684443 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:06.684470 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:06.684586 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:06.684907 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 19:44:06.685092 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":313,"threads_started":1}
I20260926 19:44:06.688956 23224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:06.689446 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:06.690102 23224 log.cc:824] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:06.693205 23224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059: No bootstrap required, opened a new log
I20260926 19:44:06.700682 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:06.694186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50451 (local address 127.0.0.1:37139)
0926 19:44:06.694353 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:06.694359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:06.694842 (+ 483us) server_negotiation.cc:407] Connection header received
0926 19:44:06.695015 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:06.695019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:06.695080 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:06.695160 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:06.695709 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:06.696477 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:06.698295 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:06.698766 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:06.698871 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:06.699981 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:06.699986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:06.699989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:06.700005 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:06.700039 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:06.700052 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:06.700136 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:06.700315 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 19:44:06.700356 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:44:06.702490 23224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff9aefb6dc74deeb2d1fed66f80b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } }
I20260926 19:44:06.702839 23224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:06.702931 23224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff9aefb6dc74deeb2d1fed66f80b059, State: Initialized, Role: FOLLOWER
I20260926 19:44:06.703516 23224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff9aefb6dc74deeb2d1fed66f80b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } }
I20260926 19:44:06.703743 23224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:06.703868 23224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:06.704049 23224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:06.706275 23224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff9aefb6dc74deeb2d1fed66f80b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } }
I20260926 19:44:06.706792 23224 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ff9aefb6dc74deeb2d1fed66f80b059; no voters:
I20260926 19:44:06.707448 23224 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:06.707564 23234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:06.708207 23234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff9aefb6dc74deeb2d1fed66f80b059, State: Running, Role: LEADER
I20260926 19:44:06.708698 23234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff9aefb6dc74deeb2d1fed66f80b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } }
I20260926 19:44:06.709407 23224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:06.711347 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ff9aefb6dc74deeb2d1fed66f80b059. Latest consensus state: current_term: 1 leader_uuid: "5ff9aefb6dc74deeb2d1fed66f80b059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff9aefb6dc74deeb2d1fed66f80b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } } }
I20260926 19:44:06.711774 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:06.712133 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ff9aefb6dc74deeb2d1fed66f80b059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff9aefb6dc74deeb2d1fed66f80b059" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37139 } } }
I20260926 19:44:06.712387 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:06.713883 23243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:06.717620 23243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:06.725085 23243 catalog_manager.cc:1347] Generated new cluster ID: 5c5d3b2cb05941ddb651c3b8a813cd12
I20260926 19:44:06.725204 23243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:06.745618 23243 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:06.746797 23243 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:06.753667 23243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ff9aefb6dc74deeb2d1fed66f80b059: Generated new TSK 0
I20260926 19:44:06.754684 23243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:06.766410 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23157
W20260926 19:44:06.767879 23219 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57639
W20260926 19:44:06.768522 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23157: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:06.768667 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 23157
W20260926 19:44:06.787367 23142 connection.cc:535] client connection to 127.0.0.1:37139 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37139): Connection reset by peer (error 104)
2026-09-26T19:44:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1752 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:06 dist-test-slave-mrw4 krb5kdc[23255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:06 dist-test-slave-mrw4 krb5kdc[23255](info): set up 2 sockets
Sep 26 19:44:06 dist-test-slave-mrw4 krb5kdc[23255](info): commencing operation
krb5kdc: starting...
W20260926 19:44:07.781235 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:07 dist-test-slave-mrw4 krb5kdc[23255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:07.994445 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:08.303160 23268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:08.303412 23268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:08.303524 23268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:08.310623 23268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:08.310721 23268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:08.310787 23268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:08.310861 23268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:08.310920 23268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:08.320327 23268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:08.321540 23268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:08.322758 23268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:08.333410 23273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:08.333431 23274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:08.333653 23276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:08.334072 23268 server_base.cc:1034] running on GCE node
I20260926 19:44:08.334758 23268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:08.336018 23268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:08.337237 23268 hybrid_clock.cc:648] HybridClock initialized: now 1790451848337182 us; error 46 us; skew 500 ppm
I20260926 19:44:08.337581 23268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:08 dist-test-slave-mrw4 krb5kdc[23255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:08.340524 23268 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:08.341889 23268 webserver.cc:460] Webserver started at http://127.0.0.1:45333/ using document root <none> and password file <none>
I20260926 19:44:08.342500 23268 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:08.342609 23268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:08.342881 23268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:08.345732 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "4759d0896a5146488fe2cc2663b9df1d"
format_stamp: "Formatted at 2026-09-26 19:44:08 on dist-test-slave-mrw4"
server_key: "529cb0ca1800ab0293a0fcbe2457bfd0"
server_key_iv: "09472adaea3a670ac15318d9404b62eb"
server_key_version: "encryptionkey@0"
I20260926 19:44:08.346426 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "4759d0896a5146488fe2cc2663b9df1d"
format_stamp: "Formatted at 2026-09-26 19:44:08 on dist-test-slave-mrw4"
server_key: "529cb0ca1800ab0293a0fcbe2457bfd0"
server_key_iv: "09472adaea3a670ac15318d9404b62eb"
server_key_version: "encryptionkey@0"
I20260926 19:44:08.351221 23268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:44:08.354307 23283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:08.355532 23268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:44:08.355674 23268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "4759d0896a5146488fe2cc2663b9df1d"
format_stamp: "Formatted at 2026-09-26 19:44:08 on dist-test-slave-mrw4"
server_key: "529cb0ca1800ab0293a0fcbe2457bfd0"
server_key_iv: "09472adaea3a670ac15318d9404b62eb"
server_key_version: "encryptionkey@0"
I20260926 19:44:08.355834 23268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:08.360937 23268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:08.364450 23268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:08.364710 23268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:08.380733 23268 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46751
I20260926 19:44:08.380774 23334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46751 every 8 connection(s)
I20260926 19:44:08.382032 23268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:08.385002 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 23268
I20260926 19:44:08.385192 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:08.385532 23335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:08.385663 2705 external_mini_cluster.cc:1428] Setting key 78b69ae0322a8128b98ad6940e7d95fa
I20260926 19:44:08.393862 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:08.387410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51113 (local address 127.0.0.1:46751)
0926 19:44:08.388311 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:08.388326 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:08.388633 (+ 307us) server_negotiation.cc:407] Connection header received
0926 19:44:08.390263 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:08.390290 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:08.390715 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:08.391139 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:08.392052 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:08.392060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:08.392077 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:08.392093 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:08.392166 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:08.392192 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:08.392417 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:08.392539 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 19:44:08.392721 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":367,"threads_started":1}
I20260926 19:44:08.397279 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:08.398710 23335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d: Bootstrap starting.
I20260926 19:44:08.401659 23335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:08.402747 23335 log.cc:824] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:08.403465 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:08.401288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35981 (local address 127.0.0.1:46751)
0926 19:44:08.401440 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:08.401446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:08.401781 (+ 335us) server_negotiation.cc:407] Connection header received
0926 19:44:08.401920 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:08.401925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:08.401989 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:08.402101 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:08.402835 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:08.402841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:08.402845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:08.402861 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:08.402903 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:08.402923 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:08.403028 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:08.403100 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 19:44:08.403153 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 19:44:08.405575 23335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d: No bootstrap required, opened a new log
I20260926 19:44:08.412525 23335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4759d0896a5146488fe2cc2663b9df1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46751 } }
I20260926 19:44:08.412865 23335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:08.412933 23335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4759d0896a5146488fe2cc2663b9df1d, State: Initialized, Role: FOLLOWER
I20260926 19:44:08.413463 23335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4759d0896a5146488fe2cc2663b9df1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46751 } }
I20260926 19:44:08.413663 23335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:08.413766 23335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:08.413875 23335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:08.416002 23335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4759d0896a5146488fe2cc2663b9df1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46751 } }
I20260926 19:44:08.416407 23335 leader_election.cc:302] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4759d0896a5146488fe2cc2663b9df1d; no voters:
I20260926 19:44:08.416873 23335 leader_election.cc:288] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:08.416976 23345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:08.417582 23345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [term 1 LEADER]: Becoming Leader. State: Replica: 4759d0896a5146488fe2cc2663b9df1d, State: Running, Role: LEADER
I20260926 19:44:08.418036 23345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4759d0896a5146488fe2cc2663b9df1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46751 } }
I20260926 19:44:08.418679 23335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:08.420694 23346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4759d0896a5146488fe2cc2663b9df1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4759d0896a5146488fe2cc2663b9df1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46751 } } }
I20260926 19:44:08.420816 23347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4759d0896a5146488fe2cc2663b9df1d. Latest consensus state: current_term: 1 leader_uuid: "4759d0896a5146488fe2cc2663b9df1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4759d0896a5146488fe2cc2663b9df1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46751 } } }
I20260926 19:44:08.421073 23346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:08.421185 23347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:08.423795 23350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:08.428557 23350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:08.436470 23350 catalog_manager.cc:1347] Generated new cluster ID: 51c2f34473894849a50517cf0a3d3c26
I20260926 19:44:08.436568 23350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:08.451632 23350 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:08.452061 23350 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:08.463239 23350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4759d0896a5146488fe2cc2663b9df1d: Generated new TSK 0
I20260926 19:44:08.464140 23350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:08.475692 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23268
W20260926 19:44:08.476672 23330 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51113
W20260926 19:44:08.477293 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23268: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:08.477418 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 23268
2026-09-26T19:44:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1702 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:08 dist-test-slave-mrw4 krb5kdc[23366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:08 dist-test-slave-mrw4 krb5kdc[23366](info): set up 2 sockets
Sep 26 19:44:08 dist-test-slave-mrw4 krb5kdc[23366](info): commencing operation
krb5kdc: starting...
W20260926 19:44:09.477241 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:09 dist-test-slave-mrw4 krb5kdc[23366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:09.693463 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:10.002272 23379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:10.002539 23379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:10.002676 23379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:10.009681 23379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:10.009778 23379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:10.009830 23379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:10.009913 23379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:10.009964 23379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:10.019446 23379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:10.020668 23379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:10.021915 23379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:10.032761 23384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:10.032845 23387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:10.032817 23385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:10.033792 23379 server_base.cc:1034] running on GCE node
I20260926 19:44:10.034487 23379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:10.035763 23379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:10.036964 23379 hybrid_clock.cc:648] HybridClock initialized: now 1790451850036929 us; error 31 us; skew 500 ppm
I20260926 19:44:10.037325 23379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:10 dist-test-slave-mrw4 krb5kdc[23366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:10.044354 23379 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:10.045858 23379 webserver.cc:460] Webserver started at http://127.0.0.1:40653/ using document root <none> and password file <none>
I20260926 19:44:10.046512 23379 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:10.046623 23379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:10.046895 23379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:10.049800 23379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "f0268f9748384c9780065066633479d3"
format_stamp: "Formatted at 2026-09-26 19:44:10 on dist-test-slave-mrw4"
server_key: "b76a073f303f6688bf13c8007fd29834"
server_key_iv: "3166be51c9209e329bb18a80e0736e33"
server_key_version: "encryptionkey@0"
I20260926 19:44:10.050504 23379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "f0268f9748384c9780065066633479d3"
format_stamp: "Formatted at 2026-09-26 19:44:10 on dist-test-slave-mrw4"
server_key: "b76a073f303f6688bf13c8007fd29834"
server_key_iv: "3166be51c9209e329bb18a80e0736e33"
server_key_version: "encryptionkey@0"
I20260926 19:44:10.055373 23379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 19:44:10.058559 23394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:10.059746 23379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 19:44:10.059903 23379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "f0268f9748384c9780065066633479d3"
format_stamp: "Formatted at 2026-09-26 19:44:10 on dist-test-slave-mrw4"
server_key: "b76a073f303f6688bf13c8007fd29834"
server_key_iv: "3166be51c9209e329bb18a80e0736e33"
server_key_version: "encryptionkey@0"
I20260926 19:44:10.060066 23379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:10.077955 23379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:10.081583 23379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:10.081835 23379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:10.098399 23379 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40773
I20260926 19:44:10.098421 23445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40773 every 8 connection(s)
I20260926 19:44:10.099692 23379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:10.102284 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 23379
I20260926 19:44:10.102460 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:10.102989 2705 external_mini_cluster.cc:1428] Setting key 9d402d151a154ca29539e22a55f8b21e
I20260926 19:44:10.103742 23446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:44:10 dist-test-slave-mrw4 krb5kdc[23366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 19:44:10.117254 23446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3: Bootstrap starting.
I20260926 19:44:10.120200 23446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:10.121377 23446 log.cc:824] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:10.124347 23446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3: No bootstrap required, opened a new log
I20260926 19:44:10.124457 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:10.104620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41713 (local address 127.0.0.1:40773)
0926 19:44:10.105350 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:10.105361 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:10.105399 (+ 38us) server_negotiation.cc:407] Connection header received
0926 19:44:10.106860 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:10.106879 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:10.107252 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:10.107557 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:10.109115 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:10.110466 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:10.112132 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:10.112898 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:10.113028 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:10.117585 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:10.117593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:10.117606 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:10.117640 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:10.120437 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:10.121288 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:10.121294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:10.121300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:10.121472 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:10.121843 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:10.121848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:10.121851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:10.122565 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:10.122800 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:10.123229 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 19:44:10.123528 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":309,"threads_started":1}
I20260926 19:44:10.127691 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:10.133458 23446 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0268f9748384c9780065066633479d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260926 19:44:10.133843 23446 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:10.133931 23446 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0268f9748384c9780065066633479d3, State: Initialized, Role: FOLLOWER
I20260926 19:44:10.134431 23446 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0268f9748384c9780065066633479d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260926 19:44:10.134617 23446 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:10.134740 23446 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:10.134879 23446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:10.137660 23446 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0268f9748384c9780065066633479d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260926 19:44:10.138212 23446 leader_election.cc:302] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0268f9748384c9780065066633479d3; no voters:
I20260926 19:44:10.138803 23446 leader_election.cc:288] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:10.138927 23456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:10.139485 23456 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [term 1 LEADER]: Becoming Leader. State: Replica: f0268f9748384c9780065066633479d3, State: Running, Role: LEADER
I20260926 19:44:10.140138 23456 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0268f9748384c9780065066633479d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260926 19:44:10.142074 23456 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0268f9748384c9780065066633479d3. Latest consensus state: current_term: 1 leader_uuid: "f0268f9748384c9780065066633479d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0268f9748384c9780065066633479d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } } }
I20260926 19:44:10.142365 23456 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:10.142786 23458 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0268f9748384c9780065066633479d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0268f9748384c9780065066633479d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } } }
I20260926 19:44:10.143100 23458 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:10.147017 23446 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:10.147475 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:10.132610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47849 (local address 127.0.0.1:40773)
0926 19:44:10.132773 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:10.132778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:10.133194 (+ 416us) server_negotiation.cc:407] Connection header received
0926 19:44:10.133314 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:10.133319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:10.133387 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:10.133498 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:10.134959 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:10.136009 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:10.137198 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:10.137829 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:10.137950 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:10.140656 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:10.140663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:10.140668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:10.140707 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:10.144600 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:10.145355 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:10.145360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:10.145364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:10.145530 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:10.145939 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:10.145944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:10.145947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:10.146332 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:10.146603 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:10.146895 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 19:44:10.147116 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:44:10.153220 23460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:10.157963 23460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:10.167203 23460 catalog_manager.cc:1347] Generated new cluster ID: ea2eedb05906444e9927c17d30b6f8a7
I20260926 19:44:10.167313 23460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:10.176077 23460 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:10.177783 23460 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:10.189481 23460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0268f9748384c9780065066633479d3: Generated new TSK 0
I20260926 19:44:10.190222 23460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:10.203598 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23379
I20260926 19:44:10.204633 23441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:10.292424 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 23379
2026-09-26T19:44:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1814 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:10 dist-test-slave-mrw4 krb5kdc[23478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:10 dist-test-slave-mrw4 krb5kdc[23478](info): set up 2 sockets
Sep 26 19:44:10 dist-test-slave-mrw4 krb5kdc[23478](info): commencing operation
krb5kdc: starting...
W20260926 19:44:11.293145 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:11 dist-test-slave-mrw4 krb5kdc[23478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:11.508862 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:11.820320 23491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:11.820600 23491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:11.820735 23491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:11.827782 23491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:11.827899 23491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:11.827976 23491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:11.828052 23491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:11.828116 23491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:11.838165 23491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:11.839393 23491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:11.840627 23491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:11.851555 23497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:11.851593 23499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:11.851635 23496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:11.852675 23491 server_base.cc:1034] running on GCE node
I20260926 19:44:11.853459 23491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:11.854753 23491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:11.855942 23491 hybrid_clock.cc:648] HybridClock initialized: now 1790451851855896 us; error 37 us; skew 500 ppm
I20260926 19:44:11.856238 23491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:11 dist-test-slave-mrw4 krb5kdc[23478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:11.859133 23491 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:11.860496 23491 webserver.cc:460] Webserver started at http://127.0.0.1:41889/ using document root <none> and password file <none>
I20260926 19:44:11.861119 23491 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:11.861272 23491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:11.861542 23491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:11.864280 23491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "7bfe4a5b517247fc97d107a4e8a81c0a"
format_stamp: "Formatted at 2026-09-26 19:44:11 on dist-test-slave-mrw4"
server_key: "5d1fe2043b5a7b7ec05aa9d01b47a09b"
server_key_iv: "26f68bed475ed996fdfae410cc6e2326"
server_key_version: "encryptionkey@0"
I20260926 19:44:11.864986 23491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "7bfe4a5b517247fc97d107a4e8a81c0a"
format_stamp: "Formatted at 2026-09-26 19:44:11 on dist-test-slave-mrw4"
server_key: "5d1fe2043b5a7b7ec05aa9d01b47a09b"
server_key_iv: "26f68bed475ed996fdfae410cc6e2326"
server_key_version: "encryptionkey@0"
I20260926 19:44:11.869674 23491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260926 19:44:11.872900 23506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:11.874152 23491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:44:11.874312 23491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "7bfe4a5b517247fc97d107a4e8a81c0a"
format_stamp: "Formatted at 2026-09-26 19:44:11 on dist-test-slave-mrw4"
server_key: "5d1fe2043b5a7b7ec05aa9d01b47a09b"
server_key_iv: "26f68bed475ed996fdfae410cc6e2326"
server_key_version: "encryptionkey@0"
I20260926 19:44:11.874472 23491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:11.909379 23491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:11.913347 23491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:11.913589 23491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:11.930617 23491 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45583
I20260926 19:44:11.930650 23557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45583 every 8 connection(s)
I20260926 19:44:11.931861 23491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:11.935267 23558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:11.940418 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 23491
I20260926 19:44:11.940595 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:11.941114 2705 external_mini_cluster.cc:1428] Setting key 7735c82e11705154ea7083fa316d8ab1
I20260926 19:44:11.945538 23558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a: Bootstrap starting.
I20260926 19:44:11.948802 23558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:11.949914 23558 log.cc:824] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:11.952785 23558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a: No bootstrap required, opened a new log
I20260926 19:44:11.953593 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:11.942872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45969 (local address 127.0.0.1:45583)
0926 19:44:11.943622 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:11.943636 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:11.943684 (+ 48us) server_negotiation.cc:407] Connection header received
0926 19:44:11.945292 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:11.945313 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:11.945697 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:11.945997 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:11.946515 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:11.947782 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:11.949532 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:11.950225 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:11.950397 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:11.951492 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:11.951499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:11.951512 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:11.952028 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:11.952102 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:11.952125 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:11.952210 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:11.952416 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 19:44:11.952623 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":342,"threads_started":1}
I20260926 19:44:11.957150 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:11.962256 23558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe4a5b517247fc97d107a4e8a81c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } }
I20260926 19:44:11.962634 23558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:11.962738 23558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bfe4a5b517247fc97d107a4e8a81c0a, State: Initialized, Role: FOLLOWER
I20260926 19:44:11.963299 23558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe4a5b517247fc97d107a4e8a81c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } }
I20260926 19:44:11.963518 23558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:11.963654 23558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:11.963817 23558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:11.965950 23558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe4a5b517247fc97d107a4e8a81c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } }
I20260926 19:44:11.966434 23558 leader_election.cc:302] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bfe4a5b517247fc97d107a4e8a81c0a; no voters:
I20260926 19:44:11.967031 23558 leader_election.cc:288] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:11.967159 23568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:11.967636 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:11.961937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45597 (local address 127.0.0.1:45583)
0926 19:44:11.962092 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:11.962098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:11.962465 (+ 367us) server_negotiation.cc:407] Connection header received
0926 19:44:11.962611 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:11.962616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:11.962676 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:11.962753 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:11.963192 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:11.963976 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:11.965272 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:11.965715 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:11.965819 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:11.966808 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:11.966814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:11.966819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:11.966843 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:11.966892 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:11.966912 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:11.967014 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:11.967204 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 19:44:11.967312 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 19:44:11.967813 23568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [term 1 LEADER]: Becoming Leader. State: Replica: 7bfe4a5b517247fc97d107a4e8a81c0a, State: Running, Role: LEADER
I20260926 19:44:11.968415 23568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe4a5b517247fc97d107a4e8a81c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } }
I20260926 19:44:11.970253 23569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bfe4a5b517247fc97d107a4e8a81c0a. Latest consensus state: current_term: 1 leader_uuid: "7bfe4a5b517247fc97d107a4e8a81c0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe4a5b517247fc97d107a4e8a81c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } } }
I20260926 19:44:11.970546 23558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:11.970609 23569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:11.970971 23570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bfe4a5b517247fc97d107a4e8a81c0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bfe4a5b517247fc97d107a4e8a81c0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } } }
I20260926 19:44:11.971297 23570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:11.973384 23572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:11.978870 23572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:11.987227 23572 catalog_manager.cc:1347] Generated new cluster ID: 345bc72d3ef949feb86389935f45ab9a
I20260926 19:44:11.987334 23572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:12.010673 23572 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:12.011960 23572 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:12.019150 23572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bfe4a5b517247fc97d107a4e8a81c0a: Generated new TSK 0
I20260926 19:44:12.019896 23572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:12.027562 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23491
W20260926 19:44:12.028561 23553 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45969
W20260926 19:44:12.029273 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23491: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:12.029403 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 23491
2026-09-26T19:44:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1736 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:12 dist-test-slave-mrw4 krb5kdc[23589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:12 dist-test-slave-mrw4 krb5kdc[23589](info): set up 2 sockets
Sep 26 19:44:12 dist-test-slave-mrw4 krb5kdc[23589](info): commencing operation
krb5kdc: starting...
W20260926 19:44:13.025238 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:13 dist-test-slave-mrw4 krb5kdc[23589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:13.240197 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:13.548336 23602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:13.548653 23602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:13.548800 23602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:13.555925 23602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:13.556037 23602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:13.556115 23602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:13.556196 23602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:13.556254 23602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:13.566033 23602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:13.567239 23602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:13.568472 23602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:13.579329 23610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:13.579324 23608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:13.579460 23607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:13.580076 23602 server_base.cc:1034] running on GCE node
I20260926 19:44:13.580740 23602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:13.582119 23602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:13.583346 23602 hybrid_clock.cc:648] HybridClock initialized: now 1790451853583297 us; error 40 us; skew 500 ppm
I20260926 19:44:13.583722 23602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:13 dist-test-slave-mrw4 krb5kdc[23589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:13.586706 23602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:13.588141 23602 webserver.cc:460] Webserver started at http://127.0.0.1:34635/ using document root <none> and password file <none>
I20260926 19:44:13.588816 23602 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:13.588917 23602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:13.589226 23602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:13.592062 23602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "398132f6c96e4e95ac7e8aef18bce2d8"
format_stamp: "Formatted at 2026-09-26 19:44:13 on dist-test-slave-mrw4"
server_key: "6c9df29e4a996b37291cdb3d7f7dc940"
server_key_iv: "5b55ccafc2c41c8cb4414616e97773f2"
server_key_version: "encryptionkey@0"
I20260926 19:44:13.592793 23602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "398132f6c96e4e95ac7e8aef18bce2d8"
format_stamp: "Formatted at 2026-09-26 19:44:13 on dist-test-slave-mrw4"
server_key: "6c9df29e4a996b37291cdb3d7f7dc940"
server_key_iv: "5b55ccafc2c41c8cb4414616e97773f2"
server_key_version: "encryptionkey@0"
I20260926 19:44:13.597658 23602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 19:44:13.600811 23617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:13.602082 23602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:44:13.602231 23602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "398132f6c96e4e95ac7e8aef18bce2d8"
format_stamp: "Formatted at 2026-09-26 19:44:13 on dist-test-slave-mrw4"
server_key: "6c9df29e4a996b37291cdb3d7f7dc940"
server_key_iv: "5b55ccafc2c41c8cb4414616e97773f2"
server_key_version: "encryptionkey@0"
I20260926 19:44:13.602393 23602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:13.607738 23602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:13.611501 23602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:13.611760 23602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:13.628064 23602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35603
I20260926 19:44:13.628103 23668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35603 every 8 connection(s)
I20260926 19:44:13.629350 23602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:13.632797 23669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:13.639293 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 23602
I20260926 19:44:13.639482 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:13.640012 2705 external_mini_cluster.cc:1428] Setting key 46b7d8b460b3411d0336f1175557e36a
I20260926 19:44:13.643277 23669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8: Bootstrap starting.
I20260926 19:44:13.646629 23669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:13.647341 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:13.641723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51733 (local address 127.0.0.1:35603)
0926 19:44:13.642438 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:13.642450 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:13.642489 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:44:13.644134 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:13.644156 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:13.644477 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:13.644797 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:13.645843 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:13.645849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:13.645861 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:13.645876 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:13.645938 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:13.645957 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:13.646083 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:13.646236 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 19:44:13.646404 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":309,"threads_started":1}
I20260926 19:44:13.647673 23669 log.cc:824] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:13.650681 23669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8: No bootstrap required, opened a new log
I20260926 19:44:13.651238 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:13.658536 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:13.656286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53413 (local address 127.0.0.1:35603)
0926 19:44:13.656464 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:13.656470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:13.656734 (+ 264us) server_negotiation.cc:407] Connection header received
0926 19:44:13.656892 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:13.656896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:13.656945 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:13.657023 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:13.657943 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:13.657949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:13.657954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:13.657970 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:13.658009 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:13.658024 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:13.658130 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:13.658221 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 19:44:13.658279 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 19:44:13.659780 23669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398132f6c96e4e95ac7e8aef18bce2d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35603 } }
I20260926 19:44:13.660100 23669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:13.660189 23669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398132f6c96e4e95ac7e8aef18bce2d8, State: Initialized, Role: FOLLOWER
I20260926 19:44:13.660796 23669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398132f6c96e4e95ac7e8aef18bce2d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35603 } }
I20260926 19:44:13.661015 23669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:13.661141 23669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:13.661319 23669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:13.663369 23669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398132f6c96e4e95ac7e8aef18bce2d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35603 } }
I20260926 19:44:13.663836 23669 leader_election.cc:302] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 398132f6c96e4e95ac7e8aef18bce2d8; no voters:
I20260926 19:44:13.664387 23669 leader_election.cc:288] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:13.664477 23679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:13.665398 23679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 398132f6c96e4e95ac7e8aef18bce2d8, State: Running, Role: LEADER
I20260926 19:44:13.665900 23679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398132f6c96e4e95ac7e8aef18bce2d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35603 } }
I20260926 19:44:13.666539 23669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:13.668473 23680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "398132f6c96e4e95ac7e8aef18bce2d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398132f6c96e4e95ac7e8aef18bce2d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35603 } } }
I20260926 19:44:13.668944 23680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:13.668481 23681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 398132f6c96e4e95ac7e8aef18bce2d8. Latest consensus state: current_term: 1 leader_uuid: "398132f6c96e4e95ac7e8aef18bce2d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398132f6c96e4e95ac7e8aef18bce2d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35603 } } }
I20260926 19:44:13.669276 23681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:13.671057 23687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:13.675997 23687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:13.684191 23687 catalog_manager.cc:1347] Generated new cluster ID: 79ccbd3eba2e43da8324ef143cb5436a
I20260926 19:44:13.684289 23687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:13.696455 23687 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:13.696892 23687 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:13.706979 23687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 398132f6c96e4e95ac7e8aef18bce2d8: Generated new TSK 0
I20260926 19:44:13.707716 23687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:13.721086 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23602
W20260926 19:44:13.722334 23664 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51733
W20260926 19:44:13.722937 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23602: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:13.723080 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 23602
2026-09-26T19:44:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1693 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:13 dist-test-slave-mrw4 krb5kdc[23700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:13 dist-test-slave-mrw4 krb5kdc[23700](info): set up 2 sockets
Sep 26 19:44:13 dist-test-slave-mrw4 krb5kdc[23700](info): commencing operation
krb5kdc: starting...
W20260926 19:44:14.717434 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:14 dist-test-slave-mrw4 krb5kdc[23700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:14.933589 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:15.242398 23713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:15.242657 23713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:15.242795 23713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:15.249763 23713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:15.249862 23713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:15.249938 23713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:15.250016 23713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:15.250075 23713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:15.259383 23713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:15.260579 23713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:15.261838 23713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:15.272804 23719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:15.272894 23718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:15.273075 23721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:15.273748 23713 server_base.cc:1034] running on GCE node
I20260926 19:44:15.274446 23713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:15.275722 23713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:15.276909 23713 hybrid_clock.cc:648] HybridClock initialized: now 1790451855276858 us; error 42 us; skew 500 ppm
I20260926 19:44:15.277228 23713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:15 dist-test-slave-mrw4 krb5kdc[23700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:15.287330 23713 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:15.288954 23713 webserver.cc:460] Webserver started at http://127.0.0.1:33947/ using document root <none> and password file <none>
I20260926 19:44:15.289677 23713 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:15.289824 23713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:15.290098 23713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:15.292977 23713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "ecb9f739646f4cf99587572b4653c714"
format_stamp: "Formatted at 2026-09-26 19:44:15 on dist-test-slave-mrw4"
server_key: "a8b5e70aefe67a033aa1ceb76b918b72"
server_key_iv: "9da1ee47c5f06f08548f12495eedb214"
server_key_version: "encryptionkey@0"
I20260926 19:44:15.293708 23713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "ecb9f739646f4cf99587572b4653c714"
format_stamp: "Formatted at 2026-09-26 19:44:15 on dist-test-slave-mrw4"
server_key: "a8b5e70aefe67a033aa1ceb76b918b72"
server_key_iv: "9da1ee47c5f06f08548f12495eedb214"
server_key_version: "encryptionkey@0"
I20260926 19:44:15.298487 23713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 19:44:15.301725 23728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:15.302927 23713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:44:15.303081 23713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "ecb9f739646f4cf99587572b4653c714"
format_stamp: "Formatted at 2026-09-26 19:44:15 on dist-test-slave-mrw4"
server_key: "a8b5e70aefe67a033aa1ceb76b918b72"
server_key_iv: "9da1ee47c5f06f08548f12495eedb214"
server_key_version: "encryptionkey@0"
I20260926 19:44:15.303246 23713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:15.320827 23713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:15.324573 23713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:15.324808 23713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:15.341459 23713 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34305
I20260926 19:44:15.341492 23779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34305 every 8 connection(s)
I20260926 19:44:15.342725 23713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:15.346282 23780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:15.352455 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 23713
I20260926 19:44:15.352599 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:15.353024 2705 external_mini_cluster.cc:1428] Setting key 829fcd20c5cc5029108be49d41bba158
I20260926 19:44:15.356199 23780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714: Bootstrap starting.
I20260926 19:44:15.359560 23780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:15.360556 23780 log.cc:824] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:15.363222 23780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714: No bootstrap required, opened a new log
Sep 26 19:44:15 dist-test-slave-mrw4 krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 19:44:15.372633 23780 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9f739646f4cf99587572b4653c714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } }
I20260926 19:44:15.373041 23780 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:15.373142 23780 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb9f739646f4cf99587572b4653c714, State: Initialized, Role: FOLLOWER
I20260926 19:44:15.373855 23780 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9f739646f4cf99587572b4653c714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } }
I20260926 19:44:15.374145 23780 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:15.374290 23780 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:15.374464 23780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:15.375519 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:15.354639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:33659 (local address 127.0.0.1:34305)
0926 19:44:15.355310 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:15.355338 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:15.355375 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:44:15.357016 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:15.357038 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:15.357454 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:15.357766 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:15.359484 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:15.360777 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:15.362408 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:15.363042 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:15.363168 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:15.367660 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:15.367670 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:15.367688 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:15.367734 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:15.371511 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:15.372218 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:15.372226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:15.372234 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:15.372433 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:15.372793 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:15.372798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:15.372801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:15.373493 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:15.373800 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:15.374113 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 19:44:15.374507 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":300,"threads_started":1}
I20260926 19:44:15.376679 23780 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9f739646f4cf99587572b4653c714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } }
I20260926 19:44:15.377130 23780 leader_election.cc:302] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecb9f739646f4cf99587572b4653c714; no voters:
I20260926 19:44:15.377720 23780 leader_election.cc:288] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:15.377827 23785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:15.379168 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:15.379122 23785 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [term 1 LEADER]: Becoming Leader. State: Replica: ecb9f739646f4cf99587572b4653c714, State: Running, Role: LEADER
I20260926 19:44:15.379557 23785 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9f739646f4cf99587572b4653c714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } }
I20260926 19:44:15.379810 23780 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:15.382016 23786 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecb9f739646f4cf99587572b4653c714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9f739646f4cf99587572b4653c714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } } }
I20260926 19:44:15.381932 23787 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecb9f739646f4cf99587572b4653c714. Latest consensus state: current_term: 1 leader_uuid: "ecb9f739646f4cf99587572b4653c714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9f739646f4cf99587572b4653c714" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34305 } } }
I20260926 19:44:15.382266 23787 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:15.382266 23786 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:15.393657 23791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:15.398274 23791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:15.409713 23791 catalog_manager.cc:1347] Generated new cluster ID: 2a5cab56383143b7b5bb506d41ff3f54
I20260926 19:44:15.409828 23791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:15.427484 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:15.404140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:42063 (local address 127.0.0.1:34305)
0926 19:44:15.404350 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:15.404356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:15.407505 (+ 3149us) server_negotiation.cc:407] Connection header received
0926 19:44:15.408256 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:15.408262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:15.408359 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:15.408465 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:15.410807 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:15.412037 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:15.413968 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:15.414669 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:15.414771 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:15.418282 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:15.418289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:15.418304 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:15.418347 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:15.423197 (+ 4850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:15.424200 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:15.424206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:15.424214 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:15.424394 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:15.424935 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:15.424941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:15.424945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:15.425394 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:15.425663 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:15.426855 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0926 19:44:15.427061 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:44:15.429008 23808 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 19:44:15.430207 23791 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:15.431489 23791 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:15.442380 23791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecb9f739646f4cf99587572b4653c714: Generated new TSK 0
I20260926 19:44:15.443095 23791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:15.448982 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23713
I20260926 19:44:15.459518 23775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:15.550443 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 23713
W20260926 19:44:15.564219 23698 connection.cc:535] client connection to 127.0.0.1:34305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34305): Connection reset by peer (error 104)
2026-09-26T19:44:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1827 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:15 dist-test-slave-mrw4 krb5kdc[23812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:15 dist-test-slave-mrw4 krb5kdc[23812](info): set up 2 sockets
Sep 26 19:44:15 dist-test-slave-mrw4 krb5kdc[23812](info): commencing operation
krb5kdc: starting...
W20260926 19:44:16.549281 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.009s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:16 dist-test-slave-mrw4 krb5kdc[23812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:16.765928 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:17.083989 23825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:17.084262 23825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:17.084407 23825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:17.091465 23825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:17.091575 23825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:17.091655 23825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:17.091733 23825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:17.091790 23825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:17.101568 23825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:17.102727 23825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:17.103940 23825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:17.114894 23833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:17.114954 23830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:17.114946 23831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:17.115773 23825 server_base.cc:1034] running on GCE node
I20260926 19:44:17.116463 23825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:17.117741 23825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:17.118965 23825 hybrid_clock.cc:648] HybridClock initialized: now 1790451857118908 us; error 45 us; skew 500 ppm
I20260926 19:44:17.119292 23825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:17 dist-test-slave-mrw4 krb5kdc[23812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:17.122258 23825 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:17.123565 23825 webserver.cc:460] Webserver started at http://127.0.0.1:39103/ using document root <none> and password file <none>
I20260926 19:44:17.124269 23825 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:17.124396 23825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:17.124672 23825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:17.127616 23825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "e9d4d08796234ca199a7a93c68cda24b"
format_stamp: "Formatted at 2026-09-26 19:44:17 on dist-test-slave-mrw4"
server_key: "87cdcd6d9f96b1dea6308f32f2326ade"
server_key_iv: "82f6d1a687a92e7fb8490379796139c1"
server_key_version: "encryptionkey@0"
I20260926 19:44:17.128333 23825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "e9d4d08796234ca199a7a93c68cda24b"
format_stamp: "Formatted at 2026-09-26 19:44:17 on dist-test-slave-mrw4"
server_key: "87cdcd6d9f96b1dea6308f32f2326ade"
server_key_iv: "82f6d1a687a92e7fb8490379796139c1"
server_key_version: "encryptionkey@0"
I20260926 19:44:17.133098 23825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260926 19:44:17.136580 23840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:17.137823 23825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 19:44:17.137970 23825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "e9d4d08796234ca199a7a93c68cda24b"
format_stamp: "Formatted at 2026-09-26 19:44:17 on dist-test-slave-mrw4"
server_key: "87cdcd6d9f96b1dea6308f32f2326ade"
server_key_iv: "82f6d1a687a92e7fb8490379796139c1"
server_key_version: "encryptionkey@0"
I20260926 19:44:17.138134 23825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:17.167485 23825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:17.171540 23825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:17.171789 23825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:17.188205 23825 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37631
I20260926 19:44:17.188232 23891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37631 every 8 connection(s)
I20260926 19:44:17.189518 23825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:17.192947 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:17.196357 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 23825
I20260926 19:44:17.196511 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:17.196954 2705 external_mini_cluster.cc:1428] Setting key ade7e747b5bc9bf48c1aa518d81840f4
I20260926 19:44:17.204700 23892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b: Bootstrap starting.
I20260926 19:44:17.208110 23892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:17.209229 23892 log.cc:824] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:17.209856 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:17.198640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:40031 (local address 127.0.0.1:37631)
0926 19:44:17.199380 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:17.199391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:17.199431 (+ 40us) server_negotiation.cc:407] Connection header received
0926 19:44:17.200969 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:17.200990 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:17.201452 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:17.201769 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:17.202324 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:17.203585 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:17.205400 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:17.206073 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:17.206237 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:17.207451 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:17.207460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:17.207472 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:17.208032 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:17.208108 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:17.208133 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:17.208324 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:17.208596 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 19:44:17.208779 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":298,"threads_started":1}
I20260926 19:44:17.211930 23892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b: No bootstrap required, opened a new log
I20260926 19:44:17.213569 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:17.221483 23892 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d4d08796234ca199a7a93c68cda24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } }
I20260926 19:44:17.222003 23892 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:17.222112 23892 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4d08796234ca199a7a93c68cda24b, State: Initialized, Role: FOLLOWER
I20260926 19:44:17.222780 23892 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d4d08796234ca199a7a93c68cda24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } }
I20260926 19:44:17.223008 23892 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:17.223140 23892 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:17.223341 23892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:17.224967 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:17.218466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:38853 (local address 127.0.0.1:37631)
0926 19:44:17.218675 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:17.218686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:17.219071 (+ 385us) server_negotiation.cc:407] Connection header received
0926 19:44:17.219250 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:17.219253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:17.219321 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:17.219406 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:17.219921 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:17.220690 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:17.222497 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:17.223023 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:17.223160 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:17.224242 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:17.224247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:17.224250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:17.224270 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:17.224311 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:17.224327 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:17.224414 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:17.224565 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 19:44:17.224612 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:44:17.226063 23892 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d4d08796234ca199a7a93c68cda24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } }
I20260926 19:44:17.226612 23892 leader_election.cc:302] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9d4d08796234ca199a7a93c68cda24b; no voters:
I20260926 19:44:17.227084 23892 leader_election.cc:288] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:17.227221 23902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:17.227823 23902 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [term 1 LEADER]: Becoming Leader. State: Replica: e9d4d08796234ca199a7a93c68cda24b, State: Running, Role: LEADER
I20260926 19:44:17.228318 23902 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d4d08796234ca199a7a93c68cda24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } }
I20260926 19:44:17.229050 23892 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:17.231204 23903 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9d4d08796234ca199a7a93c68cda24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d4d08796234ca199a7a93c68cda24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } } }
I20260926 19:44:17.231182 23904 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9d4d08796234ca199a7a93c68cda24b. Latest consensus state: current_term: 1 leader_uuid: "e9d4d08796234ca199a7a93c68cda24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d4d08796234ca199a7a93c68cda24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } } }
I20260926 19:44:17.231616 23903 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:17.231619 23904 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:17.233033 23907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:17.238080 23907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:17.247155 23907 catalog_manager.cc:1347] Generated new cluster ID: 92a515cd351f4b3b8b09ada988c5b739
I20260926 19:44:17.247244 23907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:17.255358 23907 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:17.256781 23907 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:17.271981 23907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9d4d08796234ca199a7a93c68cda24b: Generated new TSK 0
I20260926 19:44:17.272763 23907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:17.280781 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23825
W20260926 19:44:17.282163 23887 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.251:40031
W20260926 19:44:17.282788 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23825: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:17.282902 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 23825
W20260926 19:44:17.296525 23810 connection.cc:535] client connection to 127.0.0.1:37631 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37631): Connection reset by peer (error 104)
2026-09-26T19:44:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1732 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:17 dist-test-slave-mrw4 krb5kdc[23923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:17 dist-test-slave-mrw4 krb5kdc[23923](info): set up 2 sockets
Sep 26 19:44:17 dist-test-slave-mrw4 krb5kdc[23923](info): commencing operation
krb5kdc: starting...
W20260926 19:44:18.289376 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:18 dist-test-slave-mrw4 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:18.502446 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:18.814240 23936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:18.814507 23936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:18.814646 23936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:18.821547 23936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:18.821657 23936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:18.821729 23936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:18.821807 23936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:18.821866 23936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:18.831265 23936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:18.832465 23936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:18.833722 23936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:18.844728 23944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:18.844799 23941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:18.844760 23942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:18.845501 23936 server_base.cc:1034] running on GCE node
I20260926 19:44:18.846141 23936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:18.847338 23936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:18.848541 23936 hybrid_clock.cc:648] HybridClock initialized: now 1790451858848492 us; error 42 us; skew 500 ppm
I20260926 19:44:18.848835 23936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:18 dist-test-slave-mrw4 krb5kdc[23923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:18.851718 23936 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:18.852964 23936 webserver.cc:460] Webserver started at http://127.0.0.1:39245/ using document root <none> and password file <none>
I20260926 19:44:18.853601 23936 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:18.853695 23936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:18.853940 23936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:18.856681 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "235fa99449584183b575e10109a81b10"
format_stamp: "Formatted at 2026-09-26 19:44:18 on dist-test-slave-mrw4"
server_key: "e86e0665074b0657e1a28f2a0a5d6088"
server_key_iv: "5dde14f9b0ffb5f26791bc1680bc3ec6"
server_key_version: "encryptionkey@0"
I20260926 19:44:18.857409 23936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "235fa99449584183b575e10109a81b10"
format_stamp: "Formatted at 2026-09-26 19:44:18 on dist-test-slave-mrw4"
server_key: "e86e0665074b0657e1a28f2a0a5d6088"
server_key_iv: "5dde14f9b0ffb5f26791bc1680bc3ec6"
server_key_version: "encryptionkey@0"
I20260926 19:44:18.862084 23936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:44:18.865252 23951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:18.866595 23936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:44:18.866777 23936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "235fa99449584183b575e10109a81b10"
format_stamp: "Formatted at 2026-09-26 19:44:18 on dist-test-slave-mrw4"
server_key: "e86e0665074b0657e1a28f2a0a5d6088"
server_key_iv: "5dde14f9b0ffb5f26791bc1680bc3ec6"
server_key_version: "encryptionkey@0"
I20260926 19:44:18.866951 23936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:18.872449 23936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:18.876300 23936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:18.876547 23936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:18.892968 23936 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41781
I20260926 19:44:18.893023 24002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41781 every 8 connection(s)
I20260926 19:44:18.894284 23936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:18.897639 24003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:18.902957 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 23936
I20260926 19:44:18.903129 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:18.903669 2705 external_mini_cluster.cc:1428] Setting key c2442c4f2d612c7dcb88a50020774aa2
I20260926 19:44:18.907776 24003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10: Bootstrap starting.
I20260926 19:44:18.911005 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:18.905381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:38067 (local address 127.0.0.1:41781)
0926 19:44:18.906149 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:18.906160 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:18.906197 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:44:18.907740 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:18.907760 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:18.908132 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:18.908541 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:18.909626 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:18.909632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:18.909644 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:18.909657 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:18.909713 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:18.909731 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:18.909918 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:18.910006 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 19:44:18.910166 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":336,"threads_started":1}
I20260926 19:44:18.911159 24003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:18.912173 24003 log.cc:824] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:18.914292 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:18.914795 24003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10: No bootstrap required, opened a new log
I20260926 19:44:18.920508 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:18.918348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:36015 (local address 127.0.0.1:41781)
0926 19:44:18.918525 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:18.918530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:18.918843 (+ 313us) server_negotiation.cc:407] Connection header received
0926 19:44:18.918979 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:18.918983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:18.919030 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:18.919105 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:18.919926 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:18.919929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:18.919932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:18.919946 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:18.919979 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:18.919992 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:18.920085 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:18.920199 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 19:44:18.920242 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:44:18.922591 24003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "235fa99449584183b575e10109a81b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } }
I20260926 19:44:18.922922 24003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:18.923012 24003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235fa99449584183b575e10109a81b10, State: Initialized, Role: FOLLOWER
I20260926 19:44:18.923517 24003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "235fa99449584183b575e10109a81b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } }
I20260926 19:44:18.923717 24003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:18.923841 24003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:18.923979 24003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:18.926038 24003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "235fa99449584183b575e10109a81b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } }
I20260926 19:44:18.926461 24003 leader_election.cc:302] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 235fa99449584183b575e10109a81b10; no voters:
I20260926 19:44:18.926982 24003 leader_election.cc:288] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:18.927098 24013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:18.928153 24013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [term 1 LEADER]: Becoming Leader. State: Replica: 235fa99449584183b575e10109a81b10, State: Running, Role: LEADER
I20260926 19:44:18.928660 24013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "235fa99449584183b575e10109a81b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } }
I20260926 19:44:18.928968 24003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:18.931105 24015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 235fa99449584183b575e10109a81b10. Latest consensus state: current_term: 1 leader_uuid: "235fa99449584183b575e10109a81b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "235fa99449584183b575e10109a81b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } } }
I20260926 19:44:18.931202 24014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "235fa99449584183b575e10109a81b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "235fa99449584183b575e10109a81b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41781 } } }
I20260926 19:44:18.931551 24014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:18.931449 24015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:18.936717 24019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:18.941671 24019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:18.950263 24019 catalog_manager.cc:1347] Generated new cluster ID: 4bdb9eb713ed4e8dad9ebeb6a945cbf7
I20260926 19:44:18.950368 24019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:18.965547 24019 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:18.966018 24019 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:18.981323 24019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 235fa99449584183b575e10109a81b10: Generated new TSK 0
I20260926 19:44:18.982362 24019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:18.990128 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23936
W20260926 19:44:18.991192 23998 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.251:38067
W20260926 19:44:18.991854 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 23936: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:18.991978 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 23936
2026-09-26T19:44:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1708 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:19 dist-test-slave-mrw4 krb5kdc[24034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:19 dist-test-slave-mrw4 krb5kdc[24034](info): set up 2 sockets
Sep 26 19:44:19 dist-test-slave-mrw4 krb5kdc[24034](info): commencing operation
krb5kdc: starting...
W20260926 19:44:19.997316 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:20 dist-test-slave-mrw4 krb5kdc[24034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:20.220090 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:20.525425 24047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:20.525700 24047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:20.525831 24047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:20.532958 24047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:20.533069 24047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:20.533144 24047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:20.533248 24047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:20.533300 24047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:20.542876 24047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46837
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:20.544173 24047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:20.545455 24047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:20.556110 24052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:20.556137 24053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:20.556159 24055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:20.556846 24047 server_base.cc:1034] running on GCE node
I20260926 19:44:20.557562 24047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:20.558812 24047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:20.560001 24047 hybrid_clock.cc:648] HybridClock initialized: now 1790451860559955 us; error 40 us; skew 500 ppm
I20260926 19:44:20.560312 24047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:20 dist-test-slave-mrw4 krb5kdc[24034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:20.566622 24047 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:20.567958 24047 webserver.cc:460] Webserver started at http://127.0.0.1:39439/ using document root <none> and password file <none>
I20260926 19:44:20.568590 24047 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:20.568698 24047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:20.568980 24047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:20.571835 24047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "a16402ddf76f4a348419f1fa898a5dc9"
format_stamp: "Formatted at 2026-09-26 19:44:20 on dist-test-slave-mrw4"
server_key: "af5d79637fe9e4f98a47c55f50a8d4d5"
server_key_iv: "e3d25d20408c490af16f6b56533d3c57"
server_key_version: "encryptionkey@0"
I20260926 19:44:20.572548 24047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "a16402ddf76f4a348419f1fa898a5dc9"
format_stamp: "Formatted at 2026-09-26 19:44:20 on dist-test-slave-mrw4"
server_key: "af5d79637fe9e4f98a47c55f50a8d4d5"
server_key_iv: "e3d25d20408c490af16f6b56533d3c57"
server_key_version: "encryptionkey@0"
I20260926 19:44:20.577231 24047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 19:44:20.580374 24062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:20.581687 24047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:44:20.581836 24047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "a16402ddf76f4a348419f1fa898a5dc9"
format_stamp: "Formatted at 2026-09-26 19:44:20 on dist-test-slave-mrw4"
server_key: "af5d79637fe9e4f98a47c55f50a8d4d5"
server_key_iv: "e3d25d20408c490af16f6b56533d3c57"
server_key_version: "encryptionkey@0"
I20260926 19:44:20.582006 24047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:20.612638 24047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:20.616716 24047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:20.616967 24047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:20.633872 24047 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46837
I20260926 19:44:20.633926 24113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46837 every 8 connection(s)
I20260926 19:44:20.635125 24047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:20.638609 24114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:20.640626 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 24047
I20260926 19:44:20.640791 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:20.641355 2705 external_mini_cluster.cc:1428] Setting key 8577534955c3ced3a06def757a82feff
I20260926 19:44:20.652179 24114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9: Bootstrap starting.
Sep 26 19:44:20 dist-test-slave-mrw4 krb5kdc[24034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 19:44:20.655797 24114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:20.656953 24114 log.cc:824] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:20.659808 24114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9: No bootstrap required, opened a new log
I20260926 19:44:20.663914 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:20.643063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:36217 (local address 127.0.0.1:46837)
0926 19:44:20.643799 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:20.643810 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:20.643854 (+ 44us) server_negotiation.cc:407] Connection header received
0926 19:44:20.645423 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:20.645444 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:20.645824 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:20.646126 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:20.647827 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:20.649186 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:20.650979 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:20.651607 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:20.651733 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:20.656487 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:20.656495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:20.656508 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:20.656543 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:20.659714 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:20.660643 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:20.660649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:20.660655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:20.660821 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:20.661324 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:20.661327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:20.661329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:20.661838 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:20.662084 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:20.662600 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 19:44:20.662895 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":528,"thread_start_us":328,"threads_started":1}
I20260926 19:44:20.667527 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:20.668045 24114 raft_consensus.cc:348] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16402ddf76f4a348419f1fa898a5dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46837 } }
I20260926 19:44:20.668428 24114 raft_consensus.cc:374] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:20.668517 24114 raft_consensus.cc:729] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16402ddf76f4a348419f1fa898a5dc9, State: Initialized, Role: FOLLOWER
I20260926 19:44:20.669126 24114 consensus_queue.cc:260] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16402ddf76f4a348419f1fa898a5dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46837 } }
I20260926 19:44:20.669409 24114 raft_consensus.cc:388] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:20.669526 24114 raft_consensus.cc:482] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:20.669674 24114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:20.672545 24114 raft_consensus.cc:504] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16402ddf76f4a348419f1fa898a5dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46837 } }
I20260926 19:44:20.673025 24114 leader_election.cc:302] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a16402ddf76f4a348419f1fa898a5dc9; no voters:
I20260926 19:44:20.673676 24114 leader_election.cc:288] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:20.673789 24124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:20.674417 24124 raft_consensus.cc:686] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [term 1 LEADER]: Becoming Leader. State: Replica: a16402ddf76f4a348419f1fa898a5dc9, State: Running, Role: LEADER
I20260926 19:44:20.675034 24124 consensus_queue.cc:237] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16402ddf76f4a348419f1fa898a5dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46837 } }
I20260926 19:44:20.676662 24124 sys_catalog.cc:455] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a16402ddf76f4a348419f1fa898a5dc9. Latest consensus state: current_term: 1 leader_uuid: "a16402ddf76f4a348419f1fa898a5dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16402ddf76f4a348419f1fa898a5dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46837 } } }
I20260926 19:44:20.676966 24124 sys_catalog.cc:458] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:20.677418 24126 sys_catalog.cc:455] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a16402ddf76f4a348419f1fa898a5dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16402ddf76f4a348419f1fa898a5dc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46837 } } }
I20260926 19:44:20.677747 24126 sys_catalog.cc:458] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:20.681874 24114 sys_catalog.cc:564] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:20.689307 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:20.672430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:40123 (local address 127.0.0.1:46837)
0926 19:44:20.672609 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:20.672615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:20.673023 (+ 408us) server_negotiation.cc:407] Connection header received
0926 19:44:20.673144 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:20.673149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:20.673258 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:20.673367 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:20.675098 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:20.676331 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:20.678231 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:20.678966 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:20.679065 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:20.682287 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:20.682293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:20.682298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:20.682343 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:20.686025 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:20.686966 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:20.686972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:20.686976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:20.687139 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:20.687689 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:20.687695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:20.687699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:20.688084 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:20.688378 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:20.688710 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 19:44:20.688906 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:44:20.692893 24128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:20.698148 24128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:20.705718 24128 catalog_manager.cc:1347] Generated new cluster ID: a7d95c99080e4d708965457b42869ec5
I20260926 19:44:20.705821 24128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:20.716957 24128 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:20.718767 24128 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:20.729017 24128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a16402ddf76f4a348419f1fa898a5dc9: Generated new TSK 0
I20260926 19:44:20.730029 24128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:20.739914 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24047
I20260926 19:44:20.740967 24109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:20.828949 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 24047
W20260926 19:44:20.842721 24032 connection.cc:535] client connection to 127.0.0.1:46837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46837): Connection reset by peer (error 104)
2026-09-26T19:44:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1836 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:20 dist-test-slave-mrw4 krb5kdc[24146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:20 dist-test-slave-mrw4 krb5kdc[24146](info): set up 2 sockets
Sep 26 19:44:20 dist-test-slave-mrw4 krb5kdc[24146](info): commencing operation
krb5kdc: starting...
W20260926 19:44:21.837448 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:21 dist-test-slave-mrw4 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:22.064069 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:22.373572 24159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:22.373849 24159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:22.373948 24159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:22.380882 24159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:22.380978 24159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:22.381021 24159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:22.381078 24159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:22.381119 24159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:22.390457 24159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39319
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:22.391615 24159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:22.392773 24159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:22.403506 24167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:22.403554 24165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:22.403606 24164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:22.404146 24159 server_base.cc:1034] running on GCE node
I20260926 19:44:22.404857 24159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:22.406157 24159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:22.407351 24159 hybrid_clock.cc:648] HybridClock initialized: now 1790451862407304 us; error 40 us; skew 500 ppm
I20260926 19:44:22.407660 24159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:22 dist-test-slave-mrw4 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:22.410637 24159 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:22.411913 24159 webserver.cc:460] Webserver started at http://127.0.0.1:38295/ using document root <none> and password file <none>
I20260926 19:44:22.412518 24159 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:22.412629 24159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:22.412889 24159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:22.415741 24159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "f127789cb1e24c64b121c12ccc5a4f74"
format_stamp: "Formatted at 2026-09-26 19:44:22 on dist-test-slave-mrw4"
server_key: "bd6a53f76f180d259a99bada44f128cf"
server_key_iv: "c9855e1f696819fa41964d5bee49b9a5"
server_key_version: "encryptionkey@0"
I20260926 19:44:22.416489 24159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "f127789cb1e24c64b121c12ccc5a4f74"
format_stamp: "Formatted at 2026-09-26 19:44:22 on dist-test-slave-mrw4"
server_key: "bd6a53f76f180d259a99bada44f128cf"
server_key_iv: "c9855e1f696819fa41964d5bee49b9a5"
server_key_version: "encryptionkey@0"
I20260926 19:44:22.421149 24159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 19:44:22.424261 24174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:22.425477 24159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 19:44:22.425603 24159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "f127789cb1e24c64b121c12ccc5a4f74"
format_stamp: "Formatted at 2026-09-26 19:44:22 on dist-test-slave-mrw4"
server_key: "bd6a53f76f180d259a99bada44f128cf"
server_key_iv: "c9855e1f696819fa41964d5bee49b9a5"
server_key_version: "encryptionkey@0"
I20260926 19:44:22.425755 24159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:22.458463 24159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:22.462169 24159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:22.462386 24159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:22.478718 24159 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39319
I20260926 19:44:22.478758 24225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39319 every 8 connection(s)
I20260926 19:44:22.479947 24159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:22.483418 24226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:22.485373 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 24159
I20260926 19:44:22.485522 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:22.485948 2705 external_mini_cluster.cc:1428] Setting key 974079dd4532270fb0b390f06edb02e5
I20260926 19:44:22.496687 24226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74: Bootstrap starting.
I20260926 19:44:22.499286 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:22.487562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:55849 (local address 127.0.0.1:39319)
0926 19:44:22.488332 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:22.488347 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:22.488397 (+ 50us) server_negotiation.cc:407] Connection header received
0926 19:44:22.490025 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:22.490055 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:22.490551 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:22.490944 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:22.491402 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:22.493278 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:22.494542 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:22.495548 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:22.495714 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:22.496710 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:22.496721 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:22.496738 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:22.497421 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:22.497520 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:22.497556 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:22.497699 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:22.497907 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 19:44:22.498071 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":293,"threads_started":1}
I20260926 19:44:22.500272 24226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:22.501291 24226 log.cc:824] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:22.502771 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:22.504384 24226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74: No bootstrap required, opened a new log
I20260926 19:44:22.513952 24226 raft_consensus.cc:348] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127789cb1e24c64b121c12ccc5a4f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } }
I20260926 19:44:22.514401 24226 raft_consensus.cc:374] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:22.514521 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:22.507779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:43825 (local address 127.0.0.1:39319)
0926 19:44:22.507952 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:22.507958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:22.508405 (+ 447us) server_negotiation.cc:407] Connection header received
0926 19:44:22.508540 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:22.508545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:22.508621 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:22.508717 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:22.509235 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:22.510487 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:22.511814 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:22.512536 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:22.512656 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:22.513718 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:22.513725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:22.513729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:22.513750 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:22.513814 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:22.513837 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:22.513958 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:22.514156 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 19:44:22.514211 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:44:22.514504 24226 raft_consensus.cc:729] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f127789cb1e24c64b121c12ccc5a4f74, State: Initialized, Role: FOLLOWER
I20260926 19:44:22.515166 24226 consensus_queue.cc:260] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127789cb1e24c64b121c12ccc5a4f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } }
I20260926 19:44:22.515395 24226 raft_consensus.cc:388] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:22.515532 24226 raft_consensus.cc:482] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:22.515717 24226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:22.517894 24226 raft_consensus.cc:504] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127789cb1e24c64b121c12ccc5a4f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } }
I20260926 19:44:22.518348 24226 leader_election.cc:302] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f127789cb1e24c64b121c12ccc5a4f74; no voters:
I20260926 19:44:22.518867 24226 leader_election.cc:288] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:22.518989 24236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:22.519627 24236 raft_consensus.cc:686] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [term 1 LEADER]: Becoming Leader. State: Replica: f127789cb1e24c64b121c12ccc5a4f74, State: Running, Role: LEADER
I20260926 19:44:22.520208 24236 consensus_queue.cc:237] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127789cb1e24c64b121c12ccc5a4f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } }
I20260926 19:44:22.520939 24226 sys_catalog.cc:564] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:22.523228 24238 sys_catalog.cc:455] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f127789cb1e24c64b121c12ccc5a4f74. Latest consensus state: current_term: 1 leader_uuid: "f127789cb1e24c64b121c12ccc5a4f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127789cb1e24c64b121c12ccc5a4f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } } }
I20260926 19:44:22.523586 24238 sys_catalog.cc:458] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:22.524133 24237 sys_catalog.cc:455] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f127789cb1e24c64b121c12ccc5a4f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127789cb1e24c64b121c12ccc5a4f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } } }
I20260926 19:44:22.524374 24237 sys_catalog.cc:458] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:22.524500 24241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:22.530707 24241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:22.540447 24241 catalog_manager.cc:1347] Generated new cluster ID: 72780a11e9ab4f84bea0bc49870959db
I20260926 19:44:22.540548 24241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:22.554803 24241 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:22.555943 24241 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:22.567216 24241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f127789cb1e24c64b121c12ccc5a4f74: Generated new TSK 0
I20260926 19:44:22.568019 24241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:22.583320 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24159
W20260926 19:44:22.584370 24221 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.251:55849
W20260926 19:44:22.585035 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24159: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:22.585202 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 24159
W20260926 19:44:22.598918 24144 connection.cc:535] client connection to 127.0.0.1:39319 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39319): Connection reset by peer (error 104)
2026-09-26T19:44:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1756 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:22 dist-test-slave-mrw4 krb5kdc[24257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:22 dist-test-slave-mrw4 krb5kdc[24257](info): set up 2 sockets
Sep 26 19:44:22 dist-test-slave-mrw4 krb5kdc[24257](info): commencing operation
krb5kdc: starting...
W20260926 19:44:23.589484 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:23 dist-test-slave-mrw4 krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:44:23.816498 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:24.123536 24270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:24.123818 24270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:24.123907 24270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:24.131023 24270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:24.131129 24270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:24.131179 24270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:24.131237 24270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:24.131278 24270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:24.140969 24270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:24.142201 24270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:24.143404 24270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:24.154604 24275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:24.154645 24276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:24.154690 24278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:24.158632 24270 server_base.cc:1034] running on GCE node
I20260926 19:44:24.159349 24270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:24.160635 24270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:24.161827 24270 hybrid_clock.cc:648] HybridClock initialized: now 1790451864161772 us; error 43 us; skew 500 ppm
I20260926 19:44:24.162143 24270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:24 dist-test-slave-mrw4 krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:24.164893 24270 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:44:24.166263 24270 webserver.cc:460] Webserver started at http://127.0.0.1:43693/ using document root <none> and password file <none>
I20260926 19:44:24.166857 24270 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:24.166949 24270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:24.167184 24270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:24.169981 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "cb41daa8873c4c9c86cd3703eb3c7711"
format_stamp: "Formatted at 2026-09-26 19:44:24 on dist-test-slave-mrw4"
server_key: "dc0919fca075213220419088fca22af2"
server_key_iv: "004f86f096480801878d221b8ffd2398"
server_key_version: "encryptionkey@0"
I20260926 19:44:24.170651 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "cb41daa8873c4c9c86cd3703eb3c7711"
format_stamp: "Formatted at 2026-09-26 19:44:24 on dist-test-slave-mrw4"
server_key: "dc0919fca075213220419088fca22af2"
server_key_iv: "004f86f096480801878d221b8ffd2398"
server_key_version: "encryptionkey@0"
I20260926 19:44:24.175263 24270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:44:24.178368 24285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:24.179559 24270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:44:24.179697 24270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "cb41daa8873c4c9c86cd3703eb3c7711"
format_stamp: "Formatted at 2026-09-26 19:44:24 on dist-test-slave-mrw4"
server_key: "dc0919fca075213220419088fca22af2"
server_key_iv: "004f86f096480801878d221b8ffd2398"
server_key_version: "encryptionkey@0"
I20260926 19:44:24.179852 24270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:24.185200 24270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:24.188777 24270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:24.189049 24270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:24.205597 24270 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35751
I20260926 19:44:24.205646 24336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35751 every 8 connection(s)
I20260926 19:44:24.206887 24270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:24.210413 24337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:24.216200 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 24270
I20260926 19:44:24.216348 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:24.216766 2705 external_mini_cluster.cc:1428] Setting key f62333d68a5f0b180a6bbaa2d68800d8
I20260926 19:44:24.220681 24337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711: Bootstrap starting.
I20260926 19:44:24.223739 24337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:24.224062 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:24.218466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:50269 (local address 127.0.0.1:35751)
0926 19:44:24.219287 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:24.219298 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:24.219335 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:44:24.220967 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:24.220988 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:24.221354 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:24.221673 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:24.222614 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:24.222620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:24.222632 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:24.222647 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:24.222702 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:24.222724 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:24.222903 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:24.223042 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 19:44:24.223205 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":333,"threads_started":1}
I20260926 19:44:24.224943 24337 log.cc:824] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:24.227638 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:24.227802 24337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711: No bootstrap required, opened a new log
I20260926 19:44:24.233678 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:24.231577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.251:33495 (local address 127.0.0.1:35751)
0926 19:44:24.231742 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:24.231747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:24.232015 (+ 268us) server_negotiation.cc:407] Connection header received
0926 19:44:24.232163 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:24.232172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:24.232220 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:24.232302 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:24.233090 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:24.233095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:24.233100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:24.233116 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:24.233151 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:24.233188 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:24.233301 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:24.233375 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 19:44:24.233435 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 19:44:24.235495 24337 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb41daa8873c4c9c86cd3703eb3c7711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } }
I20260926 19:44:24.235812 24337 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:24.235905 24337 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb41daa8873c4c9c86cd3703eb3c7711, State: Initialized, Role: FOLLOWER
I20260926 19:44:24.236413 24337 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb41daa8873c4c9c86cd3703eb3c7711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } }
I20260926 19:44:24.236627 24337 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:24.236748 24337 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:24.236903 24337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:24.239344 24337 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb41daa8873c4c9c86cd3703eb3c7711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } }
I20260926 19:44:24.239813 24337 leader_election.cc:302] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb41daa8873c4c9c86cd3703eb3c7711; no voters:
I20260926 19:44:24.240360 24337 leader_election.cc:288] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:24.240479 24347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:24.241055 24347 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [term 1 LEADER]: Becoming Leader. State: Replica: cb41daa8873c4c9c86cd3703eb3c7711, State: Running, Role: LEADER
I20260926 19:44:24.241598 24347 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb41daa8873c4c9c86cd3703eb3c7711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } }
I20260926 19:44:24.242417 24337 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:24.244164 24348 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb41daa8873c4c9c86cd3703eb3c7711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb41daa8873c4c9c86cd3703eb3c7711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } } }
I20260926 19:44:24.244582 24348 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:24.244187 24349 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb41daa8873c4c9c86cd3703eb3c7711. Latest consensus state: current_term: 1 leader_uuid: "cb41daa8873c4c9c86cd3703eb3c7711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb41daa8873c4c9c86cd3703eb3c7711" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } } }
I20260926 19:44:24.245325 24349 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:24.246019 24354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:24.250859 24354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:24.259294 24354 catalog_manager.cc:1347] Generated new cluster ID: 5399a259a1d84a0d8a9be5e5e7dfabd8
I20260926 19:44:24.259397 24354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:24.282054 24354 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:24.282459 24354 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:24.295452 24354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb41daa8873c4c9c86cd3703eb3c7711: Generated new TSK 0
I20260926 19:44:24.296430 24354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:24.301653 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24270
W20260926 19:44:24.302678 24332 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.251:50269
W20260926 19:44:24.303333 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24270: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:24.303455 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 24270
2026-09-26T19:44:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1717 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:24 dist-test-slave-mrw4 krb5kdc[24368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:24 dist-test-slave-mrw4 krb5kdc[24368](info): set up 2 sockets
Sep 26 19:44:24 dist-test-slave-mrw4 krb5kdc[24368](info): commencing operation
krb5kdc: starting...
W20260926 19:44:25.309468 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:25 dist-test-slave-mrw4 krb5kdc[24368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:44:25.528077 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38835
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:38835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:25.835662 24381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:25.835911 24381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:25.836040 24381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:25.842990 24381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:25.843091 24381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:25.843170 24381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:25.843246 24381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:25.843300 24381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:25.852550 24381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:38835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:38835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:25.853727 24381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:25.854908 24381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:25.865729 24389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:25.865763 24387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:25.865792 24386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:25.866816 24381 server_base.cc:1034] running on GCE node
I20260926 19:44:25.867553 24381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:25.868888 24381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:25.870076 24381 hybrid_clock.cc:648] HybridClock initialized: now 1790451865870036 us; error 33 us; skew 500 ppm
I20260926 19:44:25.870388 24381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:25 dist-test-slave-mrw4 krb5kdc[24368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451865, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:25.876026 24381 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:44:25.877449 24381 webserver.cc:460] Webserver started at http://127.2.164.126:44191/ using document root <none> and password file <none>
I20260926 19:44:25.878078 24381 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:25.878191 24381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:25.878459 24381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:25.881326 24381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "5447a2bab3ca4096ae31adec76f96a0c"
format_stamp: "Formatted at 2026-09-26 19:44:25 on dist-test-slave-mrw4"
server_key: "d6aea533e35ace196f586d47cd09ef58"
server_key_iv: "1e945afdc85dea0d3cbe7e5876c72e10"
server_key_version: "encryptionkey@0"
I20260926 19:44:25.882045 24381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "5447a2bab3ca4096ae31adec76f96a0c"
format_stamp: "Formatted at 2026-09-26 19:44:25 on dist-test-slave-mrw4"
server_key: "d6aea533e35ace196f586d47cd09ef58"
server_key_iv: "1e945afdc85dea0d3cbe7e5876c72e10"
server_key_version: "encryptionkey@0"
I20260926 19:44:25.886735 24381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 19:44:25.889982 24396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:25.891175 24381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:44:25.891325 24381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "5447a2bab3ca4096ae31adec76f96a0c"
format_stamp: "Formatted at 2026-09-26 19:44:25 on dist-test-slave-mrw4"
server_key: "d6aea533e35ace196f586d47cd09ef58"
server_key_iv: "1e945afdc85dea0d3cbe7e5876c72e10"
server_key_version: "encryptionkey@0"
I20260926 19:44:25.891481 24381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:25.914263 24381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:25.918081 24381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:25.918320 24381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:25.934595 24381 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:38835
I20260926 19:44:25.934628 24447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:38835 every 8 connection(s)
I20260926 19:44:25.935787 24381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:25.938280 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 24381
I20260926 19:44:25.938444 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:25.938974 2705 external_mini_cluster.cc:1428] Setting key fc848f19c970e4334572476de723c572
I20260926 19:44:25.939675 24448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:44:25 dist-test-slave-mrw4 krb5kdc[24368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:44:25.953639 24448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c: Bootstrap starting.
I20260926 19:44:25.956673 24448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:25.957870 24448 log.cc:824] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:25.960778 24448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c: No bootstrap required, opened a new log
I20260926 19:44:25.960837 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:25.940631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47417 (local address 127.2.164.126:38835)
0926 19:44:25.941363 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:25.941374 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:25.941415 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:44:25.942990 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:25.943012 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:25.943383 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:25.943706 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:25.945402 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:25.947123 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:25.948742 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:25.949408 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:25.949560 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:25.954079 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:25.954087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:25.954103 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:25.954140 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:25.956834 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:25.957665 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:25.957671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:25.957676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:25.957822 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:25.958262 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:25.958265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:25.958267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:25.958937 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:25.959230 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:25.959621 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 19:44:25.959882 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":313,"threads_started":1}
I20260926 19:44:25.964395 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:25.968621 24448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5447a2bab3ca4096ae31adec76f96a0c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38835 } }
I20260926 19:44:25.969007 24448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:25.969120 24448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5447a2bab3ca4096ae31adec76f96a0c, State: Initialized, Role: FOLLOWER
I20260926 19:44:25.969705 24448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5447a2bab3ca4096ae31adec76f96a0c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38835 } }
I20260926 19:44:25.969904 24448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:25.970022 24448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:25.970158 24448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:25.972723 24448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5447a2bab3ca4096ae31adec76f96a0c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38835 } }
I20260926 19:44:25.973239 24448 leader_election.cc:302] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5447a2bab3ca4096ae31adec76f96a0c; no voters:
I20260926 19:44:25.973874 24448 leader_election.cc:288] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:25.973981 24458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:25.975680 24448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:25.980762 24458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [term 1 LEADER]: Becoming Leader. State: Replica: 5447a2bab3ca4096ae31adec76f96a0c, State: Running, Role: LEADER
I20260926 19:44:25.981528 24458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5447a2bab3ca4096ae31adec76f96a0c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38835 } }
I20260926 19:44:25.984196 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:25.968359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39177 (local address 127.2.164.126:38835)
0926 19:44:25.968497 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:25.968503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:25.968905 (+ 402us) server_negotiation.cc:407] Connection header received
0926 19:44:25.969065 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:25.969071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:25.969153 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:25.969269 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:25.970679 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:25.971832 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:25.972998 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:25.973651 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:25.973783 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:25.976981 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:25.976988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:25.977005 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:25.977052 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:25.980795 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:25.981764 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:25.981770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:25.981776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:25.982010 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:25.982612 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:25.982618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:25.982621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:25.982955 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:25.983268 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:25.983679 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 19:44:25.983857 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 19:44:25.984601 24459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5447a2bab3ca4096ae31adec76f96a0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5447a2bab3ca4096ae31adec76f96a0c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38835 } } }
I20260926 19:44:25.984956 24459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:25.986001 24460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5447a2bab3ca4096ae31adec76f96a0c. Latest consensus state: current_term: 1 leader_uuid: "5447a2bab3ca4096ae31adec76f96a0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5447a2bab3ca4096ae31adec76f96a0c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 38835 } } }
I20260926 19:44:25.986759 24460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c [sys.catalog]: This master's current role is: LEADER
W20260926 19:44:25.989899 24472 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:44:25.989992 24472 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:44:25.990113 24473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:25.992144 24473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:26.000034 24473 catalog_manager.cc:1347] Generated new cluster ID: 6e5aacd2d52a4442a56fda04af9d80b3
I20260926 19:44:26.000147 24473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:26.025362 24473 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:26.027139 24473 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:26.038635 24473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5447a2bab3ca4096ae31adec76f96a0c: Generated new TSK 0
I20260926 19:44:26.039484 24473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:26.044642 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24381
I20260926 19:44:26.046414 24443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:26.135973 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 24381
W20260926 19:44:26.149840 24366 connection.cc:535] client connection to 127.2.164.126:38835 recv error: Network error: failed to read from TLS socket (remote: 127.2.164.126:38835): Connection reset by peer (error 104)
2026-09-26T19:44:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1833 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:26 dist-test-slave-mrw4 krb5kdc[24480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:26 dist-test-slave-mrw4 krb5kdc[24480](info): set up 2 sockets
Sep 26 19:44:26 dist-test-slave-mrw4 krb5kdc[24480](info): commencing operation
krb5kdc: starting...
W20260926 19:44:27.153357 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:27 dist-test-slave-mrw4 krb5kdc[24480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:44:27.374239 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:37097
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:37097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:27.686053 24493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:27.686319 24493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:27.686453 24493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:27.693485 24493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:27.693584 24493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:27.693653 24493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:27.693727 24493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:27.693773 24493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:27.703321 24493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:37097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:27.704535 24493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:27.705829 24493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:27.716796 24501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:27.716766 24499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:27.717048 24498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:27.717543 24493 server_base.cc:1034] running on GCE node
I20260926 19:44:27.718220 24493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:27.719491 24493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:27.720713 24493 hybrid_clock.cc:648] HybridClock initialized: now 1790451867720659 us; error 46 us; skew 500 ppm
I20260926 19:44:27.721033 24493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:27 dist-test-slave-mrw4 krb5kdc[24480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451867, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:27.723940 24493 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:44:27.725278 24493 webserver.cc:460] Webserver started at http://127.2.164.126:38403/ using document root <none> and password file <none>
I20260926 19:44:27.725878 24493 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:27.725991 24493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:27.726263 24493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:27.729019 24493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "02eb835989724878b71919f7193feccc"
format_stamp: "Formatted at 2026-09-26 19:44:27 on dist-test-slave-mrw4"
server_key: "5fafdfa1e8968ec3b07d9ef42fa138cb"
server_key_iv: "aa04e1bddc6f0ff0ad6df6416f49c510"
server_key_version: "encryptionkey@0"
I20260926 19:44:27.729780 24493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "02eb835989724878b71919f7193feccc"
format_stamp: "Formatted at 2026-09-26 19:44:27 on dist-test-slave-mrw4"
server_key: "5fafdfa1e8968ec3b07d9ef42fa138cb"
server_key_iv: "aa04e1bddc6f0ff0ad6df6416f49c510"
server_key_version: "encryptionkey@0"
I20260926 19:44:27.734476 24493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:44:27.737728 24508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:27.738950 24493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:44:27.739117 24493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "02eb835989724878b71919f7193feccc"
format_stamp: "Formatted at 2026-09-26 19:44:27 on dist-test-slave-mrw4"
server_key: "5fafdfa1e8968ec3b07d9ef42fa138cb"
server_key_iv: "aa04e1bddc6f0ff0ad6df6416f49c510"
server_key_version: "encryptionkey@0"
I20260926 19:44:27.739282 24493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:27.759491 24493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:27.763259 24493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:27.763501 24493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:27.780207 24493 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:37097
I20260926 19:44:27.780229 24559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:37097 every 8 connection(s)
I20260926 19:44:27.781546 24493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:27.784041 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 24493
I20260926 19:44:27.784198 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:27.784602 2705 external_mini_cluster.cc:1428] Setting key 7585f58bc2bca4e99a57b4de058b12e1
I20260926 19:44:27.785122 24560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:27.796368 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:27.786217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56813 (local address 127.2.164.126:37097)
0926 19:44:27.786926 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:27.786937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:27.786979 (+ 42us) server_negotiation.cc:407] Connection header received
0926 19:44:27.788454 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:27.788473 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:27.788846 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:27.789232 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:27.789752 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:27.791043 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:27.792302 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:27.792933 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:27.793088 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:27.794316 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:27.794323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:27.794335 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:27.794881 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:27.794953 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:27.794975 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:27.795123 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:27.795384 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 19:44:27.795520 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":323,"threads_started":1}
I20260926 19:44:27.796739 24560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc: Bootstrap starting.
I20260926 19:44:27.799845 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:27.799985 24560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:27.800889 24560 log.cc:824] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:27.803272 24560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc: No bootstrap required, opened a new log
I20260926 19:44:27.810846 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:27.803820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36423 (local address 127.2.164.126:37097)
0926 19:44:27.803987 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:27.803994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:27.804478 (+ 484us) server_negotiation.cc:407] Connection header received
0926 19:44:27.804645 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:27.804648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:27.804699 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:27.804782 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:27.805374 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:27.806120 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:27.807932 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:27.808765 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:27.808917 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:27.810006 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:27.810011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:27.810017 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:27.810034 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:27.810096 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:27.810116 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:27.810227 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:27.810478 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 19:44:27.810531 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 19:44:27.812490 24560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eb835989724878b71919f7193feccc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37097 } }
I20260926 19:44:27.812799 24560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:27.812891 24560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02eb835989724878b71919f7193feccc, State: Initialized, Role: FOLLOWER
I20260926 19:44:27.813462 24560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eb835989724878b71919f7193feccc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37097 } }
I20260926 19:44:27.813669 24560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:27.813787 24560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:27.813977 24560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:27.815961 24560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eb835989724878b71919f7193feccc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37097 } }
I20260926 19:44:27.816395 24560 leader_election.cc:302] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02eb835989724878b71919f7193feccc; no voters:
I20260926 19:44:27.816895 24560 leader_election.cc:288] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:27.817003 24570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:27.817622 24570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [term 1 LEADER]: Becoming Leader. State: Replica: 02eb835989724878b71919f7193feccc, State: Running, Role: LEADER
I20260926 19:44:27.818094 24570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eb835989724878b71919f7193feccc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37097 } }
I20260926 19:44:27.818712 24560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:27.820597 24572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02eb835989724878b71919f7193feccc. Latest consensus state: current_term: 1 leader_uuid: "02eb835989724878b71919f7193feccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eb835989724878b71919f7193feccc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37097 } } }
I20260926 19:44:27.820576 24571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02eb835989724878b71919f7193feccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eb835989724878b71919f7193feccc" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 37097 } } }
I20260926 19:44:27.820976 24572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:27.820976 24571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:27.824319 24576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:27.829609 24576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:27.838068 24576 catalog_manager.cc:1347] Generated new cluster ID: 34866f229b014f30862f0604b7f95c12
I20260926 19:44:27.838163 24576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:27.856144 24576 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:27.857825 24576 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:27.866276 24576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02eb835989724878b71919f7193feccc: Generated new TSK 0
I20260926 19:44:27.867038 24576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:27.881618 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24493
W20260926 19:44:27.882568 24555 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56813
W20260926 19:44:27.883231 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24493: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:27.883347 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 24493
W20260926 19:44:27.896806 24478 connection.cc:535] client connection to 127.2.164.126:37097 recv error: Network error: failed to read from TLS socket (remote: 127.2.164.126:37097): Connection reset by peer (error 104)
2026-09-26T19:44:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1746 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:27 dist-test-slave-mrw4 krb5kdc[24591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:27 dist-test-slave-mrw4 krb5kdc[24591](info): set up 2 sockets
Sep 26 19:44:27 dist-test-slave-mrw4 krb5kdc[24591](info): commencing operation
krb5kdc: starting...
W20260926 19:44:28.889451 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:29 dist-test-slave-mrw4 krb5kdc[24591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:44:29.116693 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35189
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:35189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:29.427470 24604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:29.427723 24604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:29.427845 24604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:29.434851 24604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:29.434953 24604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:29.435024 24604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:29.435092 24604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:29.435153 24604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:29.444559 24604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:35189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:35189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:29.445758 24604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:29.446983 24604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:29.458906 24609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:29.459036 24612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:29.458937 24610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:29.460287 24604 server_base.cc:1034] running on GCE node
I20260926 19:44:29.461153 24604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:29.462457 24604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:29.463649 24604 hybrid_clock.cc:648] HybridClock initialized: now 1790451869463605 us; error 48 us; skew 500 ppm
I20260926 19:44:29.463953 24604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:29 dist-test-slave-mrw4 krb5kdc[24591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451869, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:29.466853 24604 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:44:29.468273 24604 webserver.cc:460] Webserver started at http://127.2.164.126:37839/ using document root <none> and password file <none>
I20260926 19:44:29.468881 24604 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:29.469010 24604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:29.469329 24604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:29.472105 24604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "85c831283dce48b7807f820fe5a9a8e3"
format_stamp: "Formatted at 2026-09-26 19:44:29 on dist-test-slave-mrw4"
server_key: "3348e459dc5d800a0e88e27851dd8061"
server_key_iv: "9ada3abbc4d1d11df5552eb5fb924f42"
server_key_version: "encryptionkey@0"
I20260926 19:44:29.472806 24604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "85c831283dce48b7807f820fe5a9a8e3"
format_stamp: "Formatted at 2026-09-26 19:44:29 on dist-test-slave-mrw4"
server_key: "3348e459dc5d800a0e88e27851dd8061"
server_key_iv: "9ada3abbc4d1d11df5552eb5fb924f42"
server_key_version: "encryptionkey@0"
I20260926 19:44:29.477600 24604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260926 19:44:29.480803 24619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:29.482008 24604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:44:29.482152 24604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "85c831283dce48b7807f820fe5a9a8e3"
format_stamp: "Formatted at 2026-09-26 19:44:29 on dist-test-slave-mrw4"
server_key: "3348e459dc5d800a0e88e27851dd8061"
server_key_iv: "9ada3abbc4d1d11df5552eb5fb924f42"
server_key_version: "encryptionkey@0"
I20260926 19:44:29.482314 24604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:29.487563 24604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:29.491304 24604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:29.491549 24604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:29.507745 24604 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:35189
I20260926 19:44:29.507773 24670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:35189 every 8 connection(s)
I20260926 19:44:29.509070 24604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:29.512645 24671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:29.517253 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 24604
I20260926 19:44:29.517410 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:29.517803 2705 external_mini_cluster.cc:1428] Setting key 1962ce73f677aa2024a2c8527bf7aa4b
I20260926 19:44:29.522955 24671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3: Bootstrap starting.
I20260926 19:44:29.525149 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:29.519509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37601 (local address 127.2.164.126:35189)
0926 19:44:29.520344 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:29.520356 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:29.520398 (+ 42us) server_negotiation.cc:407] Connection header received
0926 19:44:29.522062 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:29.522087 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:29.522416 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:29.522726 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:29.523727 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:29.523733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:29.523745 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:29.523756 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:29.523804 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:29.523822 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:29.523978 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:29.524073 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 19:44:29.524242 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":374,"threads_started":1}
I20260926 19:44:29.526180 24671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:29.527150 24671 log.cc:824] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:29.528484 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:29.530124 24671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3: No bootstrap required, opened a new log
I20260926 19:44:29.535681 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:29.533494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57519 (local address 127.2.164.126:35189)
0926 19:44:29.533683 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:29.533689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:29.533896 (+ 207us) server_negotiation.cc:407] Connection header received
0926 19:44:29.534029 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:29.534038 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:29.534099 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:29.534202 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:29.534968 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:29.534973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:29.534981 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:29.534999 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:29.535050 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:29.535068 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:29.535177 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:29.535327 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 19:44:29.535383 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 19:44:29.538568 24671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c831283dce48b7807f820fe5a9a8e3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35189 } }
I20260926 19:44:29.538882 24671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:29.538964 24671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c831283dce48b7807f820fe5a9a8e3, State: Initialized, Role: FOLLOWER
I20260926 19:44:29.539506 24671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c831283dce48b7807f820fe5a9a8e3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35189 } }
I20260926 19:44:29.539726 24671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:29.539848 24671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:29.539988 24671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:29.542079 24671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c831283dce48b7807f820fe5a9a8e3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35189 } }
I20260926 19:44:29.542510 24671 leader_election.cc:302] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c831283dce48b7807f820fe5a9a8e3; no voters:
I20260926 19:44:29.543030 24671 leader_election.cc:288] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:29.543192 24681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:29.545209 24671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:29.545463 24681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [term 1 LEADER]: Becoming Leader. State: Replica: 85c831283dce48b7807f820fe5a9a8e3, State: Running, Role: LEADER
I20260926 19:44:29.545958 24681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c831283dce48b7807f820fe5a9a8e3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35189 } }
I20260926 19:44:29.548799 24683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85c831283dce48b7807f820fe5a9a8e3. Latest consensus state: current_term: 1 leader_uuid: "85c831283dce48b7807f820fe5a9a8e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c831283dce48b7807f820fe5a9a8e3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35189 } } }
I20260926 19:44:29.549124 24682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85c831283dce48b7807f820fe5a9a8e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c831283dce48b7807f820fe5a9a8e3" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 35189 } } }
I20260926 19:44:29.549381 24683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:29.549449 24682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3 [sys.catalog]: This master's current role is: LEADER
W20260926 19:44:29.558110 24697 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:44:29.558207 24697 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:44:29.558337 24690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:29.560366 24690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:29.568171 24690 catalog_manager.cc:1347] Generated new cluster ID: 3b7374127f534421bd5a216bec657702
I20260926 19:44:29.568266 24690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:29.581489 24690 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:29.581888 24690 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:29.594066 24690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85c831283dce48b7807f820fe5a9a8e3: Generated new TSK 0
I20260926 19:44:29.594816 24690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:29.606345 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24604
W20260926 19:44:29.607306 24666 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37601
W20260926 19:44:29.607892 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24604: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:29.608006 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 24604
2026-09-26T19:44:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1724 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:29 dist-test-slave-mrw4 krb5kdc[24702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:29 dist-test-slave-mrw4 krb5kdc[24702](info): set up 2 sockets
Sep 26 19:44:29 dist-test-slave-mrw4 krb5kdc[24702](info): commencing operation
krb5kdc: starting...
W20260926 19:44:30.609356 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:30 dist-test-slave-mrw4 krb5kdc[24702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:44:30.824251 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44863
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:44863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:31.135705 24715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:31.135962 24715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:31.136081 24715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:31.143036 24715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:31.143137 24715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:31.143200 24715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:31.143273 24715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:31.143318 24715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:31.152843 24715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:44863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:44863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:31.154042 24715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:31.155287 24715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:31.166023 24721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:31.166080 24720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:31.166103 24723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:31.166775 24715 server_base.cc:1034] running on GCE node
I20260926 19:44:31.167451 24715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:31.168665 24715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:31.169853 24715 hybrid_clock.cc:648] HybridClock initialized: now 1790451871169804 us; error 40 us; skew 500 ppm
I20260926 19:44:31.170161 24715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:31 dist-test-slave-mrw4 krb5kdc[24702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451871, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:31.176482 24715 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:44:31.177851 24715 webserver.cc:460] Webserver started at http://127.2.164.126:39411/ using document root <none> and password file <none>
I20260926 19:44:31.178449 24715 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:31.178565 24715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:31.178826 24715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:31.181598 24715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "1ec96c0526b2439b900bfad3519e9e57"
format_stamp: "Formatted at 2026-09-26 19:44:31 on dist-test-slave-mrw4"
server_key: "f7067b54d5590e933105763bda90f1ff"
server_key_iv: "4f8447043aa36f8b66092cafdd234710"
server_key_version: "encryptionkey@0"
I20260926 19:44:31.182309 24715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "1ec96c0526b2439b900bfad3519e9e57"
format_stamp: "Formatted at 2026-09-26 19:44:31 on dist-test-slave-mrw4"
server_key: "f7067b54d5590e933105763bda90f1ff"
server_key_iv: "4f8447043aa36f8b66092cafdd234710"
server_key_version: "encryptionkey@0"
I20260926 19:44:31.186903 24715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:44:31.190063 24730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:31.191279 24715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:44:31.191426 24715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "1ec96c0526b2439b900bfad3519e9e57"
format_stamp: "Formatted at 2026-09-26 19:44:31 on dist-test-slave-mrw4"
server_key: "f7067b54d5590e933105763bda90f1ff"
server_key_iv: "4f8447043aa36f8b66092cafdd234710"
server_key_version: "encryptionkey@0"
I20260926 19:44:31.191581 24715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:31.214098 24715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:31.218104 24715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:31.218351 24715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:31.235446 24715 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:44863
I20260926 19:44:31.235464 24781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:44863 every 8 connection(s)
I20260926 19:44:31.236757 24715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:31.240322 24782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:31.243932 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 24715
I20260926 19:44:31.244107 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:31.244606 2705 external_mini_cluster.cc:1428] Setting key dd2c517eff7324b91b2f5c11f0badbd5
I20260926 19:44:31.252425 24782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57: Bootstrap starting.
I20260926 19:44:31.255941 24782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:31.257215 24782 log.cc:824] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:44:31 dist-test-slave-mrw4 krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:44:31.260166 24782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57: No bootstrap required, opened a new log
I20260926 19:44:31.266821 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:31.246326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54787 (local address 127.2.164.126:44863)
0926 19:44:31.246987 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:31.246997 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:31.247036 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:44:31.248577 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:31.248598 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:31.248998 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:31.249356 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:31.250989 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:31.252426 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:31.254170 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:31.254817 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:31.254940 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:31.259674 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:31.259682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:31.259696 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:31.259741 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:31.262860 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:31.263651 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:31.263657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:31.263663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:31.263842 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:31.264204 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:31.264207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:31.264209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:31.264732 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:31.264999 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:31.265429 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 19:44:31.265781 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":299,"threads_started":1}
I20260926 19:44:31.268663 24782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec96c0526b2439b900bfad3519e9e57" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44863 } }
I20260926 19:44:31.269019 24782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:31.269134 24782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec96c0526b2439b900bfad3519e9e57, State: Initialized, Role: FOLLOWER
I20260926 19:44:31.269695 24782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec96c0526b2439b900bfad3519e9e57" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44863 } }
I20260926 19:44:31.269922 24782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:31.270044 24782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:31.270198 24782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:31.270434 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:31.273130 24782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec96c0526b2439b900bfad3519e9e57" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44863 } }
I20260926 19:44:31.273728 24782 leader_election.cc:302] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ec96c0526b2439b900bfad3519e9e57; no voters:
I20260926 19:44:31.274362 24782 leader_election.cc:288] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:31.274421 24790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:31.276551 24782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:31.277666 24790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [term 1 LEADER]: Becoming Leader. State: Replica: 1ec96c0526b2439b900bfad3519e9e57, State: Running, Role: LEADER
I20260926 19:44:31.278244 24790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec96c0526b2439b900bfad3519e9e57" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44863 } }
I20260926 19:44:31.283890 24793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ec96c0526b2439b900bfad3519e9e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec96c0526b2439b900bfad3519e9e57" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44863 } } }
I20260926 19:44:31.283923 24792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ec96c0526b2439b900bfad3519e9e57. Latest consensus state: current_term: 1 leader_uuid: "1ec96c0526b2439b900bfad3519e9e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec96c0526b2439b900bfad3519e9e57" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 44863 } } }
I20260926 19:44:31.284315 24793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:31.284318 24792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:31.291435 24801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:31.296541 24801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:31.302488 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:31.279281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48861 (local address 127.2.164.126:44863)
0926 19:44:31.279463 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:31.279469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:31.283736 (+ 4267us) server_negotiation.cc:407] Connection header received
0926 19:44:31.284731 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:31.284738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:31.284821 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:31.284893 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:31.287116 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:31.288277 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:31.290525 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:31.291177 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:31.291306 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:31.294355 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:31.294362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:31.294367 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:31.294411 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:31.298408 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:31.299377 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:31.299383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:31.299390 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:31.299558 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:31.299989 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:31.299994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:31.299998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:31.300432 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:31.300677 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:31.301835 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0926 19:44:31.302084 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:44:31.311753 24801 catalog_manager.cc:1347] Generated new cluster ID: 86edaf442078498eb90ce9daec0b0ee5
I20260926 19:44:31.311866 24801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:31.328471 24801 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:31.330294 24801 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:31.341002 24801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ec96c0526b2439b900bfad3519e9e57: Generated new TSK 0
I20260926 19:44:31.341958 24801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:31.357590 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24715
I20260926 19:44:31.358598 24777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:31.448827 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 24715
2026-09-26T19:44:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1843 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:31 dist-test-slave-mrw4 krb5kdc[24814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:31 dist-test-slave-mrw4 krb5kdc[24814](info): set up 2 sockets
Sep 26 19:44:31 dist-test-slave-mrw4 krb5kdc[24814](info): commencing operation
krb5kdc: starting...
W20260926 19:44:32.445536 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:32 dist-test-slave-mrw4 krb5kdc[24814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:44:32.669016 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:41617
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:41617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:32.983635 24827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:32.983906 24827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:32.984042 24827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:32.991406 24827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:32.991549 24827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:32.991626 24827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:32.991708 24827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:32.991766 24827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:33.001735 24827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:41617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:41617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:33.003039 24827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:33.004315 24827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:33.015506 24832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:33.015544 24835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:33.015576 24833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:33.016566 24827 server_base.cc:1034] running on GCE node
I20260926 19:44:33.017346 24827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:33.018707 24827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:33.019929 24827 hybrid_clock.cc:648] HybridClock initialized: now 1790451873019885 us; error 36 us; skew 500 ppm
I20260926 19:44:33.020272 24827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:33 dist-test-slave-mrw4 krb5kdc[24814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451873, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:33.023437 24827 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:44:33.024915 24827 webserver.cc:460] Webserver started at http://127.2.164.126:40243/ using document root <none> and password file <none>
I20260926 19:44:33.025594 24827 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:33.025712 24827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:33.025978 24827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:33.028841 24827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "e605cb5774dc42b89c7137dffd68e51c"
format_stamp: "Formatted at 2026-09-26 19:44:33 on dist-test-slave-mrw4"
server_key: "000ccb9c6c981f0403c5cc397be25e6e"
server_key_iv: "d0d6386c226e879e152ee05e4da5d6e3"
server_key_version: "encryptionkey@0"
I20260926 19:44:33.029611 24827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "e605cb5774dc42b89c7137dffd68e51c"
format_stamp: "Formatted at 2026-09-26 19:44:33 on dist-test-slave-mrw4"
server_key: "000ccb9c6c981f0403c5cc397be25e6e"
server_key_iv: "d0d6386c226e879e152ee05e4da5d6e3"
server_key_version: "encryptionkey@0"
I20260926 19:44:33.034500 24827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 19:44:33.037897 24842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:33.039155 24827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:44:33.039306 24827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "e605cb5774dc42b89c7137dffd68e51c"
format_stamp: "Formatted at 2026-09-26 19:44:33 on dist-test-slave-mrw4"
server_key: "000ccb9c6c981f0403c5cc397be25e6e"
server_key_iv: "d0d6386c226e879e152ee05e4da5d6e3"
server_key_version: "encryptionkey@0"
I20260926 19:44:33.039481 24827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:33.060725 24827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:33.064774 24827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:33.065032 24827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:33.082087 24827 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:41617
I20260926 19:44:33.082103 24893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:41617 every 8 connection(s)
I20260926 19:44:33.083426 24827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:33.087038 24894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:33.089782 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 24827
I20260926 19:44:33.089957 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:33.090479 2705 external_mini_cluster.cc:1428] Setting key 2a26e1b646b2352e29efe61351c87444
I20260926 19:44:33.098475 24894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c: Bootstrap starting.
I20260926 19:44:33.101648 24894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:33.102707 24894 log.cc:824] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:33.102931 24897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:33.092381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43341 (local address 127.2.164.126:41617)
0926 19:44:33.093115 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:33.093127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:33.093185 (+ 58us) server_negotiation.cc:407] Connection header received
0926 19:44:33.094844 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:33.094867 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:33.095256 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:33.095609 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:33.096121 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:33.097458 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:33.098722 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:33.099378 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:33.099539 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:33.100688 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:33.100695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:33.100709 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:33.101250 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:33.101338 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:33.101363 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:33.101457 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:33.101675 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 19:44:33.101937 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":312,"threads_started":1}
I20260926 19:44:33.105504 24894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c: No bootstrap required, opened a new log
I20260926 19:44:33.106433 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:33.113489 24894 raft_consensus.cc:348] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605cb5774dc42b89c7137dffd68e51c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41617 } }
I20260926 19:44:33.113911 24894 raft_consensus.cc:374] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:33.114017 24894 raft_consensus.cc:729] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e605cb5774dc42b89c7137dffd68e51c, State: Initialized, Role: FOLLOWER
I20260926 19:44:33.114610 24894 consensus_queue.cc:260] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605cb5774dc42b89c7137dffd68e51c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41617 } }
I20260926 19:44:33.114835 24894 raft_consensus.cc:388] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:33.114966 24894 raft_consensus.cc:482] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:33.115154 24894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:33.116819 24897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:33.110389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37693 (local address 127.2.164.126:41617)
0926 19:44:33.110578 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:33.110585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:33.111005 (+ 420us) server_negotiation.cc:407] Connection header received
0926 19:44:33.111180 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:33.111185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:33.111245 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:33.111332 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:33.111911 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:33.112629 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:33.114390 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:33.114863 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:33.114979 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:33.116086 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:33.116091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:33.116094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:33.116110 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:33.116149 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:33.116175 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:33.116244 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:33.116427 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 19:44:33.116500 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:44:33.117913 24894 raft_consensus.cc:504] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605cb5774dc42b89c7137dffd68e51c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41617 } }
I20260926 19:44:33.118374 24894 leader_election.cc:302] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e605cb5774dc42b89c7137dffd68e51c; no voters:
I20260926 19:44:33.118903 24894 leader_election.cc:288] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:33.119004 24904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:33.119604 24904 raft_consensus.cc:686] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [term 1 LEADER]: Becoming Leader. State: Replica: e605cb5774dc42b89c7137dffd68e51c, State: Running, Role: LEADER
I20260926 19:44:33.120179 24904 consensus_queue.cc:237] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605cb5774dc42b89c7137dffd68e51c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41617 } }
I20260926 19:44:33.120801 24894 sys_catalog.cc:564] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:33.122653 24905 sys_catalog.cc:455] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e605cb5774dc42b89c7137dffd68e51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605cb5774dc42b89c7137dffd68e51c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41617 } } }
I20260926 19:44:33.123696 24905 sys_catalog.cc:458] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:33.124086 24906 sys_catalog.cc:455] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e605cb5774dc42b89c7137dffd68e51c. Latest consensus state: current_term: 1 leader_uuid: "e605cb5774dc42b89c7137dffd68e51c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e605cb5774dc42b89c7137dffd68e51c" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41617 } } }
I20260926 19:44:33.125257 24906 sys_catalog.cc:458] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:33.126591 24912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:33.131449 24912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:33.140065 24912 catalog_manager.cc:1347] Generated new cluster ID: 1e4734c6bd5c4019ba125af38390273c
I20260926 19:44:33.140174 24912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:33.160845 24912 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:33.162237 24912 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:33.169472 24912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e605cb5774dc42b89c7137dffd68e51c: Generated new TSK 0
I20260926 19:44:33.170248 24912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:33.176982 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24827
W20260926 19:44:33.178032 24889 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43341
W20260926 19:44:33.178673 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24827: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:33.178802 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 24827
2026-09-26T19:44:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1727 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:33 dist-test-slave-mrw4 krb5kdc[24925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:33 dist-test-slave-mrw4 krb5kdc[24925](info): set up 2 sockets
Sep 26 19:44:33 dist-test-slave-mrw4 krb5kdc[24925](info): commencing operation
krb5kdc: starting...
W20260926 19:44:34.181305 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:34 dist-test-slave-mrw4 krb5kdc[24925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:44:34.400490 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36537
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:36537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:34.707091 24938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:34.707376 24938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:34.707492 24938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:34.714556 24938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:34.714658 24938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:34.714725 24938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:34.714807 24938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:34.714864 24938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:34.724433 24938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:36537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:36537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:34.725700 24938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:34.726970 24938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:34.737807 24943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:34.737778 24944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:34.737963 24946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:34.738545 24938 server_base.cc:1034] running on GCE node
I20260926 19:44:34.739194 24938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:34.740427 24938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:34.741602 24938 hybrid_clock.cc:648] HybridClock initialized: now 1790451874741567 us; error 29 us; skew 500 ppm
I20260926 19:44:34.741887 24938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:34 dist-test-slave-mrw4 krb5kdc[24925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:34.744611 24938 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:44:34.745960 24938 webserver.cc:460] Webserver started at http://127.2.164.126:43919/ using document root <none> and password file <none>
I20260926 19:44:34.746554 24938 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:34.746665 24938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:34.746929 24938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:34.749748 24938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "2ccfe2f1490e45b8b65a8d3c5f0f7cf5"
format_stamp: "Formatted at 2026-09-26 19:44:34 on dist-test-slave-mrw4"
server_key: "5fa9d554ca177b5d4bc8a6b9b5b97100"
server_key_iv: "3c07473f73151b3ce46a5588392b302e"
server_key_version: "encryptionkey@0"
I20260926 19:44:34.750453 24938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "2ccfe2f1490e45b8b65a8d3c5f0f7cf5"
format_stamp: "Formatted at 2026-09-26 19:44:34 on dist-test-slave-mrw4"
server_key: "5fa9d554ca177b5d4bc8a6b9b5b97100"
server_key_iv: "3c07473f73151b3ce46a5588392b302e"
server_key_version: "encryptionkey@0"
I20260926 19:44:34.755085 24938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 19:44:34.758276 24953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:34.759509 24938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:44:34.759648 24938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "2ccfe2f1490e45b8b65a8d3c5f0f7cf5"
format_stamp: "Formatted at 2026-09-26 19:44:34 on dist-test-slave-mrw4"
server_key: "5fa9d554ca177b5d4bc8a6b9b5b97100"
server_key_iv: "3c07473f73151b3ce46a5588392b302e"
server_key_version: "encryptionkey@0"
I20260926 19:44:34.759815 24938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:34.765094 24938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:34.768553 24938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:34.768797 24938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:34.784937 24938 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:36537
I20260926 19:44:34.784976 25004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:36537 every 8 connection(s)
I20260926 19:44:34.786201 24938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:34.789776 25005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:34.789857 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 24938
I20260926 19:44:34.790019 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:34.790582 2705 external_mini_cluster.cc:1428] Setting key 7583ff7ee03d517761e28c939f935b2a
I20260926 19:44:34.797812 25008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:34.792321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57749 (local address 127.2.164.126:36537)
0926 19:44:34.793148 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:34.793180 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:34.793236 (+ 56us) server_negotiation.cc:407] Connection header received
0926 19:44:34.794682 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:34.794702 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:34.795023 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:34.795387 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:34.796350 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:34.796356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:34.796368 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:34.796382 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:34.796439 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:34.796460 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:34.796673 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:34.796804 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 19:44:34.796968 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":385,"threads_started":1}
I20260926 19:44:34.801412 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:34.802816 25005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5: Bootstrap starting.
I20260926 19:44:34.806483 25005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:34.807834 25005 log.cc:824] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:34.810288 25008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:34.807115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41321 (local address 127.2.164.126:36537)
0926 19:44:34.807343 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:34.807349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:34.807968 (+ 619us) server_negotiation.cc:407] Connection header received
0926 19:44:34.808250 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:34.808255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:34.808311 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:34.808417 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:34.809627 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:34.809632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:34.809636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:44:34.809656 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:34.809706 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:44:34.809727 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:34.809837 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:34.809970 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 19:44:34.810027 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 19:44:34.811414 25005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5: No bootstrap required, opened a new log
I20260926 19:44:34.818125 25005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccfe2f1490e45b8b65a8d3c5f0f7cf5" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36537 } }
I20260926 19:44:34.818449 25005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:44:34.818531 25005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccfe2f1490e45b8b65a8d3c5f0f7cf5, State: Initialized, Role: FOLLOWER
I20260926 19:44:34.819068 25005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccfe2f1490e45b8b65a8d3c5f0f7cf5" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36537 } }
I20260926 19:44:34.819280 25005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:44:34.819396 25005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:44:34.819530 25005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:34.821640 25005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccfe2f1490e45b8b65a8d3c5f0f7cf5" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36537 } }
I20260926 19:44:34.822079 25005 leader_election.cc:302] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ccfe2f1490e45b8b65a8d3c5f0f7cf5; no voters:
I20260926 19:44:34.822567 25005 leader_election.cc:288] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:44:34.822679 25015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:34.823249 25015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [term 1 LEADER]: Becoming Leader. State: Replica: 2ccfe2f1490e45b8b65a8d3c5f0f7cf5, State: Running, Role: LEADER
I20260926 19:44:34.823753 25015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccfe2f1490e45b8b65a8d3c5f0f7cf5" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36537 } }
I20260926 19:44:34.824388 25005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:34.826598 25016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ccfe2f1490e45b8b65a8d3c5f0f7cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccfe2f1490e45b8b65a8d3c5f0f7cf5" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36537 } } }
I20260926 19:44:34.826175 25017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ccfe2f1490e45b8b65a8d3c5f0f7cf5. Latest consensus state: current_term: 1 leader_uuid: "2ccfe2f1490e45b8b65a8d3c5f0f7cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccfe2f1490e45b8b65a8d3c5f0f7cf5" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 36537 } } }
I20260926 19:44:34.827564 25016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:34.827564 25017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:34.828493 25025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:34.833125 25025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:34.842139 25025 catalog_manager.cc:1347] Generated new cluster ID: e80917c8433a4028ab2eacba0b8b4e78
I20260926 19:44:34.842242 25025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:34.861763 25025 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:34.862130 25025 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:34.868988 25025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ccfe2f1490e45b8b65a8d3c5f0f7cf5: Generated new TSK 0
I20260926 19:44:34.869727 25025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:44:34.885489 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24938
W20260926 19:44:34.886432 25000 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57749
W20260926 19:44:34.887085 2705 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 24938: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:44:34.887252 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 24938
2026-09-26T19:44:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1707 ms)
[----------] 18 tests from AuthTokenIssuingTest (31729 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:34 dist-test-slave-mrw4 krb5kdc[25036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:34 dist-test-slave-mrw4 krb5kdc[25036](info): set up 2 sockets
Sep 26 19:44:34 dist-test-slave-mrw4 krb5kdc[25036](info): commencing operation
krb5kdc: starting...
W20260926 19:44:35.897516 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:36 dist-test-slave-mrw4 krb5kdc[25036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:44:36.129037 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:41179
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:41179,127.2.164.125:42401,127.2.164.124:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:36.437183 25049 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:44:36.437505 25049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:36.437600 25049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:36.437714 25049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:36.444661 25049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:36.444754 25049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:36.444834 25049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:36.444896 25049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:36.444976 25049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:36.454444 25049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:41179,127.2.164.125:42401,127.2.164.124:41373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:41179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:36.455678 25049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:36.456904 25049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:36.467780 25054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:36.467742 25055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:36.467819 25057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:36.468922 25049 server_base.cc:1034] running on GCE node
I20260926 19:44:36.469686 25049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:36.470973 25049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:36.472182 25049 hybrid_clock.cc:648] HybridClock initialized: now 1790451876472126 us; error 35 us; skew 500 ppm
I20260926 19:44:36.472476 25049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:36 dist-test-slave-mrw4 krb5kdc[25036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:36.480084 25049 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:44:36.481382 25049 webserver.cc:460] Webserver started at http://127.2.164.126:34587/ using document root <none> and password file <none>
I20260926 19:44:36.481980 25049 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:36.482069 25049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:36.482321 25049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:36.485036 25049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "5e05d5d5064341459163ce7c61182b21"
format_stamp: "Formatted at 2026-09-26 19:44:36 on dist-test-slave-mrw4"
server_key: "16a00ba69fb97b6267c88aa6a11aa285"
server_key_iv: "4851b41edc9f8d32e9a7998854b52f99"
server_key_version: "encryptionkey@0"
I20260926 19:44:36.485775 25049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "5e05d5d5064341459163ce7c61182b21"
format_stamp: "Formatted at 2026-09-26 19:44:36 on dist-test-slave-mrw4"
server_key: "16a00ba69fb97b6267c88aa6a11aa285"
server_key_iv: "4851b41edc9f8d32e9a7998854b52f99"
server_key_version: "encryptionkey@0"
I20260926 19:44:36.490446 25049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:44:36.493502 25064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:36.494709 25049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 19:44:36.494853 25049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "5e05d5d5064341459163ce7c61182b21"
format_stamp: "Formatted at 2026-09-26 19:44:36 on dist-test-slave-mrw4"
server_key: "16a00ba69fb97b6267c88aa6a11aa285"
server_key_iv: "4851b41edc9f8d32e9a7998854b52f99"
server_key_version: "encryptionkey@0"
I20260926 19:44:36.495028 25049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:36.513960 25049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:36.518034 25049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:36.518287 25049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:36.535181 25049 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:41179
I20260926 19:44:36.535203 25115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:41179 every 8 connection(s)
I20260926 19:44:36.536403 25049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:36.539485 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 25049
I20260926 19:44:36.539686 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:36.540268 25116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:36.540470 2705 external_mini_cluster.cc:1428] Setting key 3c8a218cb59351484de2a08c8b3088af
I20260926 19:44:36.548417 25116 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:44:36 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.125@KRBTEST.COM" created.
I20260926 19:44:36.569516 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.551872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.2.164.126:41179)
0926 19:44:36.552706 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:36.552724 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:36.552910 (+ 186us) server_negotiation.cc:407] Connection header received
0926 19:44:36.554779 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:36.554784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:36.554986 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:36.555171 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:36.556340 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:36.557539 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:36.558822 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:36.559342 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:36.559511 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:36.562608 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:36.562619 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:36.562633 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:36.562676 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:36.565776 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:36.566408 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:36.566414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:36.566416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:36.566586 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:36.566977 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:36.566981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:36.566984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:36.567465 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:36.567619 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:36.568211 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 19:44:36.568460 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"spinlock_wait_cycles":64896,"thread_start_us":447,"threads_started":1}
I20260926 19:44:36.569540 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.551872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41179 (local address 127.0.0.1:37498)
0926 19:44:36.552775 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:36.552808 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:36.554447 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:36.555195 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:36.555201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:36.555849 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:36.556150 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:36.556166 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:36.557667 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:36.557670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:36.558689 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:36.558700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:36.559545 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:36.559552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:36.559716 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:36.560274 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:36.560296 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:36.562377 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:36.565900 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:36.565910 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:36.565919 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:36.566282 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:36.566702 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:36.566706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:36.566708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:36.566871 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:36.567604 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:36.567612 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:36.567848 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 19:44:36.568116 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 19:44:36.568627 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":58,"thread_start_us":351,"threads_started":1}
I20260926 19:44:36.572263 25116 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:44:36.573299 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.572837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:35598)
0926 19:44:36.573006 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:36.573119 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 19:44:36.573526 25065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
I20260926 19:44:36.574900 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.574575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:35604)
0926 19:44:36.574707 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:36.574779 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 19:44:36.575065 25116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:42401: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab.
Entry for principal kudu/127.2.164.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab.
I20260926 19:44:36.595844 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.125:42401
--webserver_interface=127.2.164.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab
--principal=kudu/127.2.164.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:41179,127.2.164.125:42401,127.2.164.124:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:44:36.624374 25116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } attempt: 1
I20260926 19:44:36.625569 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.625049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:35608)
0926 19:44:36.625270 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:36.625405 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 19:44:36.626546 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.626158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:35624)
0926 19:44:36.626289 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:36.626369 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 19:44:36.626730 25116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:42401: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
I20260926 19:44:36.695001 25116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } attempt: 2
I20260926 19:44:36.696204 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.695686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:35628)
0926 19:44:36.695901 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:36.696022 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 19:44:36.697371 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.696892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:35644)
0926 19:44:36.697054 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:36.697202 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 19:44:36.697574 25116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:42401: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
I20260926 19:44:36.788846 25116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } attempt: 3
I20260926 19:44:36.790126 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.789574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:36572)
0926 19:44:36.789828 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:36.789950 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 19:44:36.791234 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.790815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:36588)
0926 19:44:36.790971 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:36.791067 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 19:44:36.791455 25116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:42401: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
W20260926 19:44:36.906574 25123 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:44:36.906870 25123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:36.906970 25123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:36.907078 25123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:36.917675 25123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:36.917758 25123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:36.917819 25123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:36.917888 25123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:36.917970 25123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:36.932137 25123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:41179,127.2.164.125:42401,127.2.164.124:41373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab
--principal=kudu/127.2.164.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.125:42401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.164.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:36.933849 25123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:36.934677 25116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } attempt: 4
I20260926 19:44:36.935539 25123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 19:44:36.935957 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.935365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:36590)
0926 19:44:36.935683 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:36.935788 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260926 19:44:36.937062 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:36.936660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:36606)
0926 19:44:36.936814 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:36.936910 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 19:44:36.937253 25116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:42401: Network error: Client connection negotiation failed: client connection to 127.2.164.125:42401: connect: Connection refused (error 111)
W20260926 19:44:36.948760 25129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:36.948794 25128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:36.949414 25131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:36.949781 25123 server_base.cc:1034] running on GCE node
I20260926 19:44:36.950477 25123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:36.951735 25123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:36.952930 25123 hybrid_clock.cc:648] HybridClock initialized: now 1790451876952855 us; error 53 us; skew 500 ppm
I20260926 19:44:36.953275 25123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:36 dist-test-slave-mrw4 krb5kdc[25036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:36.956215 25123 init.cc:377] Logged in from keytab as kudu/127.2.164.125@KRBTEST.COM (short username kudu)
I20260926 19:44:36.957536 25123 webserver.cc:460] Webserver started at http://127.2.164.125:45759/ using document root <none> and password file <none>
I20260926 19:44:36.958154 25123 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:36.958251 25123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:36.958527 25123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:36.961304 25123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/data/instance:
uuid: "4e3af71094014574a60cba263cf570b4"
format_stamp: "Formatted at 2026-09-26 19:44:36 on dist-test-slave-mrw4"
server_key: "15ffcea8f7b622b6031790ca58330d93"
server_key_iv: "baf60b019e42df088709fa57c74b3bcc"
server_key_version: "encryptionkey@0"
I20260926 19:44:36.962018 25123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/wal/instance:
uuid: "4e3af71094014574a60cba263cf570b4"
format_stamp: "Formatted at 2026-09-26 19:44:36 on dist-test-slave-mrw4"
server_key: "15ffcea8f7b622b6031790ca58330d93"
server_key_iv: "baf60b019e42df088709fa57c74b3bcc"
server_key_version: "encryptionkey@0"
I20260926 19:44:36.966794 25123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:44:36.969874 25138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:36.971091 25123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 19:44:36.971235 25123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/wal
uuid: "4e3af71094014574a60cba263cf570b4"
format_stamp: "Formatted at 2026-09-26 19:44:36 on dist-test-slave-mrw4"
server_key: "15ffcea8f7b622b6031790ca58330d93"
server_key_iv: "baf60b019e42df088709fa57c74b3bcc"
server_key_version: "encryptionkey@0"
I20260926 19:44:36.971410 25123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:37.003504 25123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:37.007565 25123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:37.007813 25123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:37.024731 25123 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.125:42401
I20260926 19:44:37.024762 25189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.125:42401 every 8 connection(s)
I20260926 19:44:37.025982 25123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
I20260926 19:44:37.027232 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 25123
I20260926 19:44:37.027400 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-1/wal/instance
I20260926 19:44:37.027943 2705 external_mini_cluster.cc:1428] Setting key 3fd5e482dd9c089c293dbae0721927b9
I20260926 19:44:37.030297 25190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:37.038476 25190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.124@KRBTEST.COM" created.
I20260926 19:44:37.059333 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.040928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.2.164.126:41179)
0926 19:44:37.041135 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.041142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.042358 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 19:44:37.044656 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.044660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.044729 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.044814 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.046719 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.047404 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.049483 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.049900 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.049996 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.053854 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.053858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.053861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.053894 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.056380 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.057214 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.057220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.057223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.057375 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.057800 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.057804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.057806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.058074 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.058183 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.058933 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.059105 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:44:37.060554 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.041429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41179 (local address 127.0.0.1:38992)
0926 19:44:37.042181 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.042226 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:37.044365 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:37.045042 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:37.045053 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:37.045760 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:37.046484 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.046507 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.047544 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.047549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.049339 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.049353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.050114 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.050121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.050397 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.051073 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:37.051103 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:37.053625 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:37.056510 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.056519 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.056535 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.057049 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.057480 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.057485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.057489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.057703 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.058179 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:37.058187 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:37.058579 (+ 392us) client_negotiation.cc:769] Sending connection context
0926 19:44:37.058847 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 19:44:37.059339 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":289,"threads_started":1}
I20260926 19:44:37.062271 25190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
I20260926 19:44:37.075448 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.063022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:36608)
0926 19:44:37.063186 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.063206 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:37.063314 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:37.064247 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:37.064251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:37.064649 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:37.064769 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.064777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.066134 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.066138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.066944 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.066954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.067697 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.067702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.067883 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.068390 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:37.068405 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:37.070285 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:37.073052 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.073057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.073061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.073480 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.073807 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.073812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.073816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.073948 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.074528 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:37.074533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:37.074664 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 19:44:37.074779 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 19:44:37.075040 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 19:44:37.075519 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.063304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.2.164.125:42401)
0926 19:44:37.063875 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.063881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.063905 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:44:37.063944 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.063950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.064112 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.064232 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.064929 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.066010 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.067064 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.067571 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.067675 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.070471 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.070480 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.070492 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.070521 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.072941 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.073593 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.073599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.073602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.073713 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.074038 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.074043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.074046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.074417 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.074593 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.074904 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.075152 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":426,"thread_start_us":294,"threads_started":1}
I20260926 19:44:37.077684 25190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:44:37.078683 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.078280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46894)
0926 19:44:37.078428 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.078547 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 19:44:37.078964 25139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
I20260926 19:44:37.080386 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.080040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46898)
0926 19:44:37.080182 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.080280 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 19:44:37.080559 25190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:41373: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab.
Entry for principal kudu/127.2.164.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab.
I20260926 19:44:37.086323 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.124:41373
--webserver_interface=127.2.164.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab
--principal=kudu/127.2.164.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:41179,127.2.164.125:42401,127.2.164.124:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:44:37.129824 25190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } attempt: 1
I20260926 19:44:37.130952 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.130519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46906)
0926 19:44:37.130697 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.130800 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 19:44:37.131913 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.131598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46908)
0926 19:44:37.131735 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.131812 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 19:44:37.132124 25190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:41373: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
I20260926 19:44:37.200310 25190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } attempt: 2
I20260926 19:44:37.201465 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.201006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46914)
0926 19:44:37.201215 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.201312 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 19:44:37.202503 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.202111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46918)
0926 19:44:37.202265 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.202359 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 19:44:37.202760 25190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:41373: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
I20260926 19:44:37.236537 25116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } attempt: 5
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
I20260926 19:44:37.250495 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.237440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.2.164.125:42401)
0926 19:44:37.237635 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.237640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.237685 (+ 45us) server_negotiation.cc:407] Connection header received
0926 19:44:37.237931 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.237936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.237999 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.238071 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.239137 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.239836 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.241550 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.242006 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.242101 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.245585 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.245590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.245593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.245624 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.248021 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.248756 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.248759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.248762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.248874 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.249286 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.249289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.249291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.249549 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.249656 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.250121 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.250282 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 19:44:37.250638 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.237350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:36618)
0926 19:44:37.237590 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.237614 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:37.237789 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:37.238141 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:37.238146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:37.238697 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:37.238972 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.238987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.239979 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.239986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.241376 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.241393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.242157 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.242164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.242371 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.242957 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:37.242980 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:37.245407 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:37.248151 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.248157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.248161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.248635 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.248972 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.248977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.248981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.249156 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.249666 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:37.249672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:37.249857 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 19:44:37.250041 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 19:44:37.250310 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 19:44:37.251647 25116 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:44:37.252585 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.252242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46920)
0926 19:44:37.252383 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.252453 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 19:44:37.253484 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.253201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46926)
0926 19:44:37.253318 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.253393 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 19:44:37.253669 25116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:41373: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
I20260926 19:44:37.294008 25190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } attempt: 3
I20260926 19:44:37.295150 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.294660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46942)
0926 19:44:37.294896 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.294987 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 19:44:37.296366 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.295895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46954)
0926 19:44:37.296154 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.296243 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260926 19:44:37.296609 25190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:41373: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
I20260926 19:44:37.304860 25116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } attempt: 1
I20260926 19:44:37.305895 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.305464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46966)
0926 19:44:37.305642 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.305740 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 19:44:37.306795 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.306459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46974)
0926 19:44:37.306624 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.306695 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 19:44:37.306996 25116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:41373: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
I20260926 19:44:37.375208 25116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } attempt: 2
I20260926 19:44:37.376345 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.375889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46980)
0926 19:44:37.376098 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.376185 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 19:44:37.377279 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.376953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46992)
0926 19:44:37.377069 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.377146 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 19:44:37.377486 25116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:41373: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
W20260926 19:44:37.404026 25197 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:44:37.404369 25197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:37.404459 25197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:37.404553 25197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:37.411549 25197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:37.411623 25197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:37.411681 25197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:37.411747 25197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:37.411814 25197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:37.421190 25197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:41179,127.2.164.125:42401,127.2.164.124:41373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab
--principal=kudu/127.2.164.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.124:41373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.164.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:37.422379 25197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:37.423616 25197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:37.434492 25202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:37.434552 25203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:37.434552 25205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:37.435369 25197 server_base.cc:1034] running on GCE node
I20260926 19:44:37.436029 25197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:37.437317 25197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:37.438524 25197 hybrid_clock.cc:648] HybridClock initialized: now 1790451877438463 us; error 40 us; skew 500 ppm
I20260926 19:44:37.438822 25197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:44:37.439806 25190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } attempt: 4
I20260926 19:44:37.440797 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.440378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:46996)
0926 19:44:37.440575 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.440652 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451877, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:37.441752 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.441424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:47002)
0926 19:44:37.441566 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.441643 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 19:44:37.441987 25190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:41373: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
I20260926 19:44:37.442183 25197 init.cc:377] Logged in from keytab as kudu/127.2.164.124@KRBTEST.COM (short username kudu)
I20260926 19:44:37.443455 25197 webserver.cc:460] Webserver started at http://127.2.164.124:36371/ using document root <none> and password file <none>
I20260926 19:44:37.444087 25197 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:37.444175 25197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:37.444459 25197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:37.447304 25197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/data/instance:
uuid: "6f18089712e04c709a65d98ae2a6321b"
format_stamp: "Formatted at 2026-09-26 19:44:37 on dist-test-slave-mrw4"
server_key: "96904d88968e81fef33b73736a4ca064"
server_key_iv: "a2bc530f76f69f1a6b4bd740538510e5"
server_key_version: "encryptionkey@0"
I20260926 19:44:37.448024 25197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/wal/instance:
uuid: "6f18089712e04c709a65d98ae2a6321b"
format_stamp: "Formatted at 2026-09-26 19:44:37 on dist-test-slave-mrw4"
server_key: "96904d88968e81fef33b73736a4ca064"
server_key_iv: "a2bc530f76f69f1a6b4bd740538510e5"
server_key_version: "encryptionkey@0"
I20260926 19:44:37.452992 25197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:44:37.457418 25212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:37.458909 25197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:44:37.459043 25197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/wal
uuid: "6f18089712e04c709a65d98ae2a6321b"
format_stamp: "Formatted at 2026-09-26 19:44:37 on dist-test-slave-mrw4"
server_key: "96904d88968e81fef33b73736a4ca064"
server_key_iv: "a2bc530f76f69f1a6b4bd740538510e5"
server_key_version: "encryptionkey@0"
I20260926 19:44:37.459234 25197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:37.480618 25197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:37.483666 25116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } attempt: 3
I20260926 19:44:37.484423 25197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:37.484586 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.484150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:47004)
0926 19:44:37.484335 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.484414 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 19:44:37.484652 25197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:37.485558 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.485156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:47006)
0926 19:44:37.485336 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.485428 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 19:44:37.485735 25116 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:41373: Network error: Client connection negotiation failed: client connection to 127.2.164.124:41373: connect: Connection refused (error 111)
I20260926 19:44:37.500595 25197 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.124:41373
I20260926 19:44:37.500613 25263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.124:41373 every 8 connection(s)
I20260926 19:44:37.501830 25197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
I20260926 19:44:37.505314 25264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:37.507237 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 25197
I20260926 19:44:37.507396 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790451540687502-2705-0/minicluster-data/master-2/wal/instance
I20260926 19:44:37.507803 2705 external_mini_cluster.cc:1428] Setting key bcba67a2bca4abd4d911595940668a4e
I20260926 19:44:37.512234 25264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451877, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:44:37.529422 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.514974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39004 (local address 127.2.164.126:41179)
0926 19:44:37.515123 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.515127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.515790 (+ 663us) server_negotiation.cc:407] Connection header received
0926 19:44:37.515958 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.515962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.516026 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.516103 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.517421 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.518167 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.519809 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.520261 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.520368 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.523553 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.523559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.523564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.523603 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.526511 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.527238 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.527243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.527273 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.527416 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.527805 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.527810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.527813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.528116 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.528307 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.528813 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.529021 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:44:37.530289 25268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.515027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41179 (local address 127.0.0.1:39004)
0926 19:44:37.515662 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.515694 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:37.515846 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:37.516186 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:37.516193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:37.516937 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:37.517273 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.517286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.518335 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.518339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.519659 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.519673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.520488 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.520496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.520675 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.521245 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:37.521280 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:37.523373 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:37.526624 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.526634 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.526645 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.527135 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.527517 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.527522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.527525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.527692 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.528249 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:37.528256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:37.528472 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 19:44:37.528717 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 19:44:37.529309 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":166,"thread_start_us":315,"threads_started":1}
I20260926 19:44:37.530311 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.509655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.2.164.124:41373)
0926 19:44:37.510506 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.510516 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.510558 (+ 42us) server_negotiation.cc:407] Connection header received
0926 19:44:37.512142 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.512162 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.512535 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.512854 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.514579 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.516388 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.517980 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.518795 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.518921 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.523147 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.523154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.523165 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.523197 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.526345 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.527062 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.527068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.527070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.527232 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.527571 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.527574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.527576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.528149 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.528416 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.528716 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.529192 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":644,"thread_start_us":397,"threads_started":1}
I20260926 19:44:37.531813 25264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451877, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:44:37.545658 25268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.532562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:42401 (local address 127.0.0.1:36632)
0926 19:44:37.532691 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.532708 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:37.532816 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:37.533143 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:37.533147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:37.533502 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:37.533627 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.533635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.534558 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.534563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.535877 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.535890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.536816 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.536822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.537025 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.537550 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:37.537566 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:37.539819 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:37.543162 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.543167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.543171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.543504 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.543959 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.543964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.543968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.544139 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.544615 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:37.544620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:37.544897 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 19:44:37.545079 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 19:44:37.545372 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 19:44:37.545737 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.532542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.2.164.125:42401)
0926 19:44:37.532712 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.532718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.532755 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:44:37.532920 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.532926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.533004 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.533092 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.533780 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.534430 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.536021 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.536663 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.536783 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.540026 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.540033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.540037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.540076 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.543047 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.543663 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.543668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.543671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.543837 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.544248 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.544254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.544256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.544502 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.544611 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.545224 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.545397 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:44:37.546615 25264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:44:37.548583 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.533087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.2.164.126:41179)
0926 19:44:37.533294 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.533299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.533370 (+ 71us) server_negotiation.cc:407] Connection header received
0926 19:44:37.533505 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.533510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.533566 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.533638 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.534803 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.535889 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.537140 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.537791 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.537902 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.541998 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.542005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.542009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.542044 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.545397 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.546123 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.546128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.546131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.546291 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.546689 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.546693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.546695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.546963 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.547211 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.548241 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.548388 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451877, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
I20260926 19:44:37.559458 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.547261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.2.164.124:41373)
0926 19:44:37.547390 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.547396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.547412 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:44:37.547455 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.547460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.547522 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.547621 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.548150 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.548859 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.550376 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.550934 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.551063 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.554194 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.554200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.554212 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.554246 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.557099 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.557725 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.557730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.557733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.557869 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.558266 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.558270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.558271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.558535 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.558637 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.559036 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.559183 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 19:44:37.559847 25268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.547163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:47024)
0926 19:44:37.547294 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.547310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:37.547386 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:37.547626 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:37.547629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:37.547930 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:37.548048 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.548055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.548995 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.549000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.550235 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.550251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.551068 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.551072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.551253 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.551723 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:37.551745 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:37.554017 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:37.557269 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.557273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.557275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.557611 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.557978 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.557983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.557987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.558143 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.558654 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:37.558660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:37.558814 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 19:44:37.558978 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 19:44:37.559351 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":80}
I20260926 19:44:37.565408 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.550159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.2.164.125:42401)
0926 19:44:37.550288 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.550292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.550306 (+ 14us) server_negotiation.cc:407] Connection header received
0926 19:44:37.550393 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.550397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.550445 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.550510 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.551784 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.552803 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.554190 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.554604 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.554716 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.559470 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.559479 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.559484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.559815 (+ 331us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.562800 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.563542 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.563545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.563547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.563701 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.564142 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.564148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.564151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.564403 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.564643 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.564888 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.565091 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 19:44:37.565403 25264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b: Bootstrap starting.
I20260926 19:44:37.568792 25264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:37.569896 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:37.569905 25264 log.cc:824] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:37.572394 25264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b: No bootstrap required, opened a new log
I20260926 19:44:37.572671 25091 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5e05d5d5064341459163ce7c61182b21"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39018
I20260926 19:44:37.582722 25264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } }
I20260926 19:44:37.583104 25264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f18089712e04c709a65d98ae2a6321b, State: Initialized, Role: FOLLOWER
I20260926 19:44:37.583563 25264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } }
I20260926 19:44:37.584391 25270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } } }
I20260926 19:44:37.584698 25270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:37.585436 25264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [sys.catalog]: configured and running, proceeding with master startup.
W20260926 19:44:37.594352 25281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:44:37.594478 25281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:44:37.592198 25197 master_runner.cc:186] Error getting master registration for 127.2.164.126:41179: OK, instance_id { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" instance_seqno: 1790451876530460 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:44:37.601526 25197 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 19:44:37.662936 25116 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } attempt: 4
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
I20260926 19:44:37.679097 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.663759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.2.164.124:41373)
0926 19:44:37.664101 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.664107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.664149 (+ 42us) server_negotiation.cc:407] Connection header received
0926 19:44:37.664224 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.664229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.664310 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.664432 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.665192 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.666324 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.668097 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.668826 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.668949 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.672570 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.672577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.672582 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.672621 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.676187 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.676802 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.676807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.676811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.676973 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.677401 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.677406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.677410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.677840 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.678013 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.678474 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.678697 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 19:44:37.679239 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.663695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:47036)
0926 19:44:37.663945 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.663965 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:37.664094 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:37.664472 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:37.664477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:37.664845 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:37.665009 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.665018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.666479 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.666484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.667946 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.667963 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.668958 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.668964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.669183 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.669826 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:37.669848 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:37.672392 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:37.676332 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.676336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.676339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.676699 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.677102 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.677106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.677108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.677301 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.677973 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:37.677977 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:37.678115 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 19:44:37.678357 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 19:44:37.678739 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 19:44:37.685011 25116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21: Bootstrap starting.
I20260926 19:44:37.688630 25116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:37.689729 25116 log.cc:824] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:37.692175 25116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21: No bootstrap required, opened a new log
I20260926 19:44:37.699357 25116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } }
I20260926 19:44:37.699735 25116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e05d5d5064341459163ce7c61182b21, State: Initialized, Role: FOLLOWER
I20260926 19:44:37.700213 25116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } }
I20260926 19:44:37.701107 25284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } } }
I20260926 19:44:37.701508 25284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:37.702203 25116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:37.702358 25091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 19:44:37.702549 25091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:37.705881 25091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } }
I20260926 19:44:37.707381 25091 leader_election.cc:288] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [CANDIDATE]: Term 1 election: Requested vote from peers 4e3af71094014574a60cba263cf570b4 (127.2.164.125:42401), 6f18089712e04c709a65d98ae2a6321b (127.2.164.124:41373)
I20260926 19:44:37.708606 25239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e05d5d5064341459163ce7c61182b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6f18089712e04c709a65d98ae2a6321b"
I20260926 19:44:37.709079 25239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:37.713922 25239 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e05d5d5064341459163ce7c61182b21 in term 1.
I20260926 19:44:37.714666 25065 leader_election.cc:302] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e05d5d5064341459163ce7c61182b21, 6f18089712e04c709a65d98ae2a6321b; no voters:
I20260926 19:44:37.715176 25284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:37.717275 25284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [term 1 LEADER]: Becoming Leader. State: Replica: 5e05d5d5064341459163ce7c61182b21, State: Running, Role: LEADER
I20260926 19:44:37.717833 25284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } }
I20260926 19:44:37.728530 25295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e05d5d5064341459163ce7c61182b21. Latest consensus state: current_term: 1 leader_uuid: "5e05d5d5064341459163ce7c61182b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } } }
I20260926 19:44:37.728922 25295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:37.730010 25299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260926 19:44:37.736632 25218 tablet.cc:2369] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:44:37.741117 25190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } attempt: 5
I20260926 19:44:37.744611 25299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 19:44:37 dist-test-slave-mrw4 krb5kdc[25036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451876, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
I20260926 19:44:37.758625 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.741665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:41373 (local address 127.0.0.1:47052)
0926 19:44:37.742019 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:37.742039 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:37.742162 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:37.743043 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:37.743049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:37.743595 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:37.743813 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.743827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.745309 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.745314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.746631 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:37.746646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.747781 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.747787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:37.747983 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.748653 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:37.748675 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:37.751468 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:37.755651 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.755657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.755662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.756190 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.756664 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:37.756670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:37.756674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:37.756839 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:37.757613 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:37.757628 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:37.757779 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 19:44:37.757919 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 19:44:37.758211 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260926 19:44:37.759944 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.742376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.2.164.124:41373)
0926 19:44:37.742545 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.742551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.742568 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:44:37.742640 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.742645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.742724 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.742825 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.743998 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.745106 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.746793 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.747430 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.747551 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.751694 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.751701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.751706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.751746 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.755485 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.756327 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.756332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.756336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.756515 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.756972 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.756978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.756982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.757329 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.757471 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.759378 (+ 1907us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.759604 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:44:37.766081 25239 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [term 1 FOLLOWER]: Refusing update from remote peer 5e05d5d5064341459163ce7c61182b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:44:37.766952 25295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:44:37.768888 25190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4: Bootstrap starting.
W20260926 19:44:37.770865 25070 tablet.cc:2369] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:44:37.772759 25190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:37.764154 25049 master_runner.cc:186] Error getting master registration for 127.2.164.125:42401: OK, instance_id { permanent_uuid: "4e3af71094014574a60cba263cf570b4" instance_seqno: 1790451877019887 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:44:37.775420 25049 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 19:44:37.782759 25190 log.cc:824] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:37.791271 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.732643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.2.164.125:42401)
0926 19:44:37.732822 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.732830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.743483 (+ 10653us) server_negotiation.cc:407] Connection header received
0926 19:44:37.759536 (+ 16053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.759542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.759613 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.759676 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.769025 (+ 9349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.770149 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.771854 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.772490 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.772600 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.782466 (+ 9866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.782473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.782478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.782518 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.786116 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.786903 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.786909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.786912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.787100 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.787541 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.787546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.787548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.787936 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.788211 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.788766 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.790911 (+ 2145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 19:44:37.794500 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.743938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.2.164.124:41373)
0926 19:44:37.756812 (+ 12874us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.756818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.761730 (+ 4912us) server_negotiation.cc:407] Connection header received
0926 19:44:37.761780 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.761784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.761847 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.761934 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.766071 (+ 4137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.767219 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.774925 (+ 7706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.775601 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.775720 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.780578 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.780586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.780592 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.780657 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.787345 (+ 6688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.788968 (+ 1623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.789001 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.789006 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.789346 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.790127 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.790132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.790166 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.790854 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.792962 (+ 2108us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.793289 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.793750 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12748,"thread_start_us":12879,"threads_started":1}
I20260926 19:44:37.796656 25270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e05d5d5064341459163ce7c61182b21. Latest consensus state: current_term: 1 leader_uuid: "5e05d5d5064341459163ce7c61182b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } } }
I20260926 19:44:37.796937 25270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:37.798611 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:37.739990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.2.164.126:41179)
0926 19:44:37.740166 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:37.740173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:37.756231 (+ 16058us) server_negotiation.cc:407] Connection header received
0926 19:44:37.757194 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:37.757200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:37.757280 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:37.757348 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:37.763616 (+ 6268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.764822 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.768991 (+ 4169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:37.769765 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:37.769881 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:37.780165 (+ 10284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:37.780172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:37.780177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:37.780231 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:37.790593 (+ 10362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.792153 (+ 1560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.792159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.792163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.792328 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:37.792874 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:37.792880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:37.792884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:37.793262 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:37.797810 (+ 4548us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:37.797947 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 19:44:37.798263 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:44:37.799542 25308 mvcc.cc:205] Tried to move back new op lower bound from 7333690891317092352 to 7333690891139211264. Current Snapshot: MvccSnapshot[applied={T|T < 7333690891317092352}]
I20260926 19:44:37.800156 25190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4: No bootstrap required, opened a new log
I20260926 19:44:37.811054 25190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } }
I20260926 19:44:37.811446 25190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3af71094014574a60cba263cf570b4, State: Initialized, Role: FOLLOWER
I20260926 19:44:37.811926 25190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } }
I20260926 19:44:37.813496 25270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e05d5d5064341459163ce7c61182b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } } }
I20260926 19:44:37.813740 25270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:37.813289 25314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } } }
I20260926 19:44:37.814160 25314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:37.814276 25190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:37.814363 25165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e05d5d5064341459163ce7c61182b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4e3af71094014574a60cba263cf570b4"
I20260926 19:44:37.814795 25165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:37.818426 25165 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e05d5d5064341459163ce7c61182b21 in term 1.
I20260926 19:44:37.819232 25164 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [term 1 FOLLOWER]: Refusing update from remote peer 5e05d5d5064341459163ce7c61182b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:44:37.819284 25284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e05d5d5064341459163ce7c61182b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } } }
I20260926 19:44:37.819574 25284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:37.821964 25295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:44:37.826408 25299 catalog_manager.cc:1347] Generated new cluster ID: 2356becc20ed4e9f8d25121019ac7569
I20260926 19:44:37.826493 25299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:37.871819 25314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e05d5d5064341459163ce7c61182b21. Latest consensus state: current_term: 1 leader_uuid: "5e05d5d5064341459163ce7c61182b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } } }
I20260926 19:44:37.872136 25314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:37.872612 25317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e05d5d5064341459163ce7c61182b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } } }
I20260926 19:44:37.872988 25317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:37.897327 25295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e05d5d5064341459163ce7c61182b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e05d5d5064341459163ce7c61182b21" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41179 } } peers { permanent_uuid: "4e3af71094014574a60cba263cf570b4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 42401 } } peers { permanent_uuid: "6f18089712e04c709a65d98ae2a6321b" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 41373 } } }
I20260926 19:44:37.897576 25295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:37.903283 25329 catalog_manager.cc:1259] Loaded cluster ID: 2356becc20ed4e9f8d25121019ac7569
I20260926 19:44:37.903446 25329 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4: loading cluster ID for follower catalog manager: success
W20260926 19:44:37.907021 25329 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260926 19:44:37.907236 25329 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:44:37.908918 25299 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:37.910876 25299 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:37.939702 25299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e05d5d5064341459163ce7c61182b21: Generated new TSK 0
I20260926 19:44:37.940877 25299 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 19:44:38.091311 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.075658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.2.164.125:42401)
0926 19:44:38.075879 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.075885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.087559 (+ 11674us) server_negotiation.cc:407] Connection header received
0926 19:44:38.089777 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.089784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.089876 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.089952 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.091076 (+ 1124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36662: BlockingRecv error: recv got EOF from 127.0.0.1:36662 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 19:44:38.095397 25120 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.083754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.2.164.126:41179)
0926 19:44:38.083946 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.083953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.092826 (+ 8873us) server_negotiation.cc:407] Connection header received
0926 19:44:38.093941 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.093947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.094051 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.094126 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.095159 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39028: BlockingRecv error: recv got EOF from 127.0.0.1:39028 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 19:44:38.096727 25305 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.088090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.2.164.124:41373)
0926 19:44:38.088249 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.088255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.093499 (+ 5244us) server_negotiation.cc:407] Connection header received
0926 19:44:38.093637 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.093642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.093721 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.093808 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.096508 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47070: BlockingRecv error: recv got EOF from 127.0.0.1:47070 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 19:44:38.109400 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.107376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.2.164.125:42401)
0926 19:44:38.107535 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.107540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.107964 (+ 424us) server_negotiation.cc:407] Connection header received
0926 19:44:38.108153 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.108159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.108281 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.108393 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.109080 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36678: BlockingRecv error: recv got EOF from 127.0.0.1:36678 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 19:44:38.118911 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.113451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.2.164.125:42401)
0926 19:44:38.113610 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.113615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.117616 (+ 4001us) server_negotiation.cc:407] Connection header received
0926 19:44:38.117807 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.117813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.117894 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.117988 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.118710 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36680: BlockingRecv error: recv got EOF from 127.0.0.1:36680 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 19:44:38.128691 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.126777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.2.164.125:42401)
0926 19:44:38.126940 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.126946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.127427 (+ 481us) server_negotiation.cc:407] Connection header received
0926 19:44:38.127579 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.127585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.127693 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.127793 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.128460 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36694: BlockingRecv error: recv got EOF from 127.0.0.1:36694 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 19:44:38.140691 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.138780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36710 (local address 127.2.164.125:42401)
0926 19:44:38.138980 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.138985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.139404 (+ 419us) server_negotiation.cc:407] Connection header received
0926 19:44:38.139568 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.139574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.139661 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.139760 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.140449 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36710: BlockingRecv error: recv got EOF from 127.0.0.1:36710 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260926 19:44:38.153539 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.151603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36716 (local address 127.2.164.125:42401)
0926 19:44:38.151770 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.151775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.152201 (+ 426us) server_negotiation.cc:407] Connection header received
0926 19:44:38.152364 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.152370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.152448 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.152553 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.153323 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36716: BlockingRecv error: recv got EOF from 127.0.0.1:36716 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 19:44:38.174630 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.172596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.2.164.125:42401)
0926 19:44:38.172769 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.172778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.173267 (+ 489us) server_negotiation.cc:407] Connection header received
0926 19:44:38.173467 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.173473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.173574 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.173688 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.174393 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36724: BlockingRecv error: recv got EOF from 127.0.0.1:36724 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 19:44:38.217547 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.215277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.2.164.125:42401)
0926 19:44:38.215457 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.215463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.215990 (+ 527us) server_negotiation.cc:407] Connection header received
0926 19:44:38.216180 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.216187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.216289 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.216421 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.217278 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36734: BlockingRecv error: recv got EOF from 127.0.0.1:36734 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 19:44:38.289134 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.286745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.2.164.125:42401)
0926 19:44:38.286951 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.286958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.287540 (+ 582us) server_negotiation.cc:407] Connection header received
0926 19:44:38.287808 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.287815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.287911 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.288014 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.288892 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36750: BlockingRecv error: recv got EOF from 127.0.0.1:36750 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 19:44:38.422243 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.420489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.2.164.125:42401)
0926 19:44:38.420729 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.420737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.420967 (+ 230us) server_negotiation.cc:407] Connection header received
0926 19:44:38.421122 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.421127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.421224 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.421310 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.422020 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36764: BlockingRecv error: recv got EOF from 127.0.0.1:36764 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 19:44:38.596441 25281 catalog_manager.cc:1259] Loaded cluster ID: 2356becc20ed4e9f8d25121019ac7569
I20260926 19:44:38.596544 25281 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b: loading cluster ID for follower catalog manager: success
I20260926 19:44:38.599349 25281 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b: acquiring CA information for follower catalog manager: success
I20260926 19:44:38.601032 25281 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6f18089712e04c709a65d98ae2a6321b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 19:44:38.684559 25194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:38.682788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.2.164.125:42401)
0926 19:44:38.682999 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:38.683005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:38.683248 (+ 243us) server_negotiation.cc:407] Connection header received
0926 19:44:38.683446 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:38.683452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:38.683527 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:38.683614 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:38.684347 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36774: BlockingRecv error: recv got EOF from 127.0.0.1:36774 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:44:38.910573 25329 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4: acquiring CA information for follower catalog manager: success
I20260926 19:44:38.912552 25329 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e3af71094014574a60cba263cf570b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:44:39.205821 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:39.200229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.2.164.125:42401)
0926 19:44:39.200680 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:39.200685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:39.200704 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:44:39.200871 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:39.200876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:39.201023 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:39.201130 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:44:39.201557 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:39.202304 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:39.203713 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:39.204105 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:39.204198 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:39.204677 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:44:39.205227 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:44:39.205375 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:39.205446 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 19:44:39.205512 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":239,"threads_started":1}
I20260926 19:44:39.217152 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:39.211461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47084 (local address 127.2.164.124:41373)
0926 19:44:39.212462 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:39.212467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:39.212482 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:44:39.212565 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:39.212568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:39.212704 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:39.212836 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:44:39.213157 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:39.213874 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:39.215198 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:39.215641 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:39.215728 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:39.216150 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:44:39.216658 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:44:39.216766 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:39.216875 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 19:44:39.216935 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":788,"threads_started":1}
I20260926 19:44:39.229578 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:39.222700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.2.164.125:42401)
0926 19:44:39.222902 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:39.222909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:39.223709 (+ 800us) server_negotiation.cc:407] Connection header received
0926 19:44:39.223931 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:39.223937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:39.224022 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:39.224105 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:44:39.224739 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:39.225674 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:39.227749 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:39.228346 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:39.228446 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:39.228881 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:44:39.228983 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:44:39.229047 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:39.229247 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 19:44:39.229321 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:44:39.230209 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:39.222691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.2.164.126:41179)
0926 19:44:39.223136 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:39.223142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:39.223819 (+ 677us) server_negotiation.cc:407] Connection header received
0926 19:44:39.224079 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:39.224085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:39.224338 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:39.224512 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:44:39.225051 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:39.225884 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:39.227568 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:39.228143 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:39.228264 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:39.228740 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:44:39.229092 (+ 352us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:44:39.229253 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:39.229892 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 19:44:39.229976 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":226,"threads_started":1}
I20260926 19:44:39.230271 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:39.224121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.2.164.124:41373)
0926 19:44:39.224278 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:39.224282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:39.224346 (+ 64us) server_negotiation.cc:407] Connection header received
0926 19:44:39.224593 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:39.224598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:39.224672 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:39.224727 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:44:39.225401 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:39.226321 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:39.228199 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:39.228734 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:39.228848 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:39.229569 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:44:39.229666 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:44:39.229760 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:39.229948 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 19:44:39.230022 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:44:39.234695 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 25049
I20260926 19:44:39.235603 25111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:39.343353 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 25049
I20260926 19:44:39.360069 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 25123
I20260926 19:44:39.361109 25185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:39.467674 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 25123
I20260926 19:44:39.483664 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 25197
I20260926 19:44:39.484743 25259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:39.588903 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 25197
2026-09-26T19:44:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4705 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:44:39 dist-test-slave-mrw4 krb5kdc[25416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:44:39 dist-test-slave-mrw4 krb5kdc[25416](info): set up 2 sockets
Sep 26 19:44:39 dist-test-slave-mrw4 krb5kdc[25416](info): commencing operation
krb5kdc: starting...
W20260926 19:44:40.605605 2705 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/test-user.keytab.
Sep 26 19:44:40 dist-test-slave-mrw4 krb5kdc[25416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:44:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.164.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
Entry for principal kudu/127.2.164.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab.
I20260926 19:44:40.826436 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:41673
--webserver_interface=127.2.164.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:41673,127.2.164.125:34627,127.2.164.124:39719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:44:41.134245 25429 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:44:41.134534 25429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:41.134627 25429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:41.134722 25429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:41.141691 25429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:41.141765 25429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:41.141839 25429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:41.141916 25429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:41.141997 25429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:41.151357 25429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:41673,127.2.164.125:34627,127.2.164.124:39719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.126.keytab
--principal=kudu/127.2.164.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.126:41673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.164.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:41.152529 25429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:41.154037 25429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:41.164734 25434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:41.164709 25437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:41.164666 25435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:41.165360 25429 server_base.cc:1034] running on GCE node
I20260926 19:44:41.166013 25429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:41.167244 25429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:41.168455 25429 hybrid_clock.cc:648] HybridClock initialized: now 1790451881168397 us; error 34 us; skew 500 ppm
I20260926 19:44:41.168746 25429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:41 dist-test-slave-mrw4 krb5kdc[25416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451881, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:41.171589 25429 init.cc:377] Logged in from keytab as kudu/127.2.164.126@KRBTEST.COM (short username kudu)
I20260926 19:44:41.172849 25429 webserver.cc:460] Webserver started at http://127.2.164.126:37307/ using document root <none> and password file <none>
I20260926 19:44:41.173492 25429 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:41.173585 25429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:41.173869 25429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:41.176553 25429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/data/instance:
uuid: "bc4bcaba08bd466aa223499cf2334809"
format_stamp: "Formatted at 2026-09-26 19:44:41 on dist-test-slave-mrw4"
server_key: "5c78a56c8f66b830ee10f3142899500c"
server_key_iv: "4eb6eb0c10f8733cec8dbc6027e81b42"
server_key_version: "encryptionkey@0"
I20260926 19:44:41.177299 25429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/wal/instance:
uuid: "bc4bcaba08bd466aa223499cf2334809"
format_stamp: "Formatted at 2026-09-26 19:44:41 on dist-test-slave-mrw4"
server_key: "5c78a56c8f66b830ee10f3142899500c"
server_key_iv: "4eb6eb0c10f8733cec8dbc6027e81b42"
server_key_version: "encryptionkey@0"
I20260926 19:44:41.181814 25429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:44:41.184809 25444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:41.186069 25429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:44:41.186205 25429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/wal
uuid: "bc4bcaba08bd466aa223499cf2334809"
format_stamp: "Formatted at 2026-09-26 19:44:41 on dist-test-slave-mrw4"
server_key: "5c78a56c8f66b830ee10f3142899500c"
server_key_iv: "4eb6eb0c10f8733cec8dbc6027e81b42"
server_key_version: "encryptionkey@0"
I20260926 19:44:41.186384 25429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:41.196946 25429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:41.200233 25429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:41.200448 25429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:41.216784 25429 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.126:41673
I20260926 19:44:41.216802 25495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.126:41673 every 8 connection(s)
I20260926 19:44:41.218093 25429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/data/info.pb
I20260926 19:44:41.221592 25496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:41.225355 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 25429
I20260926 19:44:41.225529 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-0/wal/instance
I20260926 19:44:41.226069 2705 external_mini_cluster.cc:1428] Setting key 76528f46a54c921ac43ad93e02b37a26
I20260926 19:44:41.229032 25496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:44:41 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451881, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.164.125@KRBTEST.COM; defaulting to no policy
I20260926 19:44:41.251194 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.232422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.2.164.126:41673)
0926 19:44:41.233226 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:41.233240 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:41.233752 (+ 512us) server_negotiation.cc:407] Connection header received
0926 19:44:41.235632 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:41.235638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:41.235857 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:41.236030 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:41.237416 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.238580 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.240382 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.240896 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.241016 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:41.244681 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:41.244688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:41.244699 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:41.244731 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:41.247321 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:41.248131 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:41.248137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:41.248139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:41.248274 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:41.248674 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:41.248677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:41.248678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:41.249104 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:41.249326 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:41.249901 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 19:44:41.250166 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":558,"thread_start_us":340,"threads_started":1}
I20260926 19:44:41.251224 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.232422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41673 (local address 127.0.0.1:60144)
0926 19:44:41.233571 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.233618 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:41.235297 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:41.236059 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:41.236066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:41.236766 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:41.237208 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:41.237233 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.238739 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.238743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:41.240246 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:41.240261 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.241206 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.241216 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:41.241377 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:41.242002 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:41.242033 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:41.244476 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:41.247458 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:41.247471 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:41.247483 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:41.248010 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:41.248378 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:41.248384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:41.248387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:41.248583 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:41.249276 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:41.249283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:41.249530 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 19:44:41.249794 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 19:44:41.250310 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":67,"spinlock_wait_cycles":16256,"thread_start_us":686,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.125@KRBTEST.COM" created.
I20260926 19:44:41.253875 25496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:44:41.254792 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.254436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36744)
0926 19:44:41.254590 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.254699 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 19:44:41.254995 25445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
I20260926 19:44:41.256336 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.256025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36748)
0926 19:44:41.256162 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.256220 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 19:44:41.256496 25496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:34627: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab.
Entry for principal kudu/127.2.164.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab.
I20260926 19:44:41.279582 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.125:34627
--webserver_interface=127.2.164.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab
--principal=kudu/127.2.164.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:41673,127.2.164.125:34627,127.2.164.124:39719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:44:41.305791 25496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } attempt: 1
I20260926 19:44:41.306895 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.306443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36750)
0926 19:44:41.306645 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.306762 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 19:44:41.307848 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.307538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36754)
0926 19:44:41.307650 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.307715 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 19:44:41.308008 25496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:34627: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
I20260926 19:44:41.376237 25496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } attempt: 2
I20260926 19:44:41.377450 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.376946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36762)
0926 19:44:41.377207 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.377315 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 19:44:41.378420 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.378094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36768)
0926 19:44:41.378232 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.378303 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 19:44:41.378608 25496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:34627: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
I20260926 19:44:41.469844 25496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } attempt: 3
I20260926 19:44:41.471179 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.470572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36776)
0926 19:44:41.470901 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.471013 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 19:44:41.472252 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.471902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36780)
0926 19:44:41.472041 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.472114 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 19:44:41.472509 25496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:34627: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
W20260926 19:44:41.588100 25503 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:44:41.588421 25503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:41.588517 25503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:41.588621 25503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:41.595746 25503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:41.595826 25503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:41.595886 25503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:41.595954 25503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:41.596030 25503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:41.605384 25503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:41673,127.2.164.125:34627,127.2.164.124:39719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.125.keytab
--principal=kudu/127.2.164.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.125:34627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.164.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:41.606542 25503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:41.607755 25503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 19:44:41.615713 25496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } attempt: 4
I20260926 19:44:41.616839 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.616283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36784)
0926 19:44:41.616560 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.616667 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260926 19:44:41.618191 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.617707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36796)
0926 19:44:41.617887 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.617990 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 19:44:41.618230 25509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:41.618312 25511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:41.618553 25496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.125:34627: Network error: Client connection negotiation failed: client connection to 127.2.164.125:34627: connect: Connection refused (error 111)
W20260926 19:44:41.618825 25508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:41.619936 25503 server_base.cc:1034] running on GCE node
I20260926 19:44:41.620692 25503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:41.621956 25503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:41.623169 25503 hybrid_clock.cc:648] HybridClock initialized: now 1790451881623106 us; error 34 us; skew 500 ppm
I20260926 19:44:41.623497 25503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:44:41 dist-test-slave-mrw4 krb5kdc[25416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451881, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:41.630256 25503 init.cc:377] Logged in from keytab as kudu/127.2.164.125@KRBTEST.COM (short username kudu)
I20260926 19:44:41.631536 25503 webserver.cc:460] Webserver started at http://127.2.164.125:45103/ using document root <none> and password file <none>
I20260926 19:44:41.632150 25503 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:41.632253 25503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:41.632531 25503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:41.635294 25503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/data/instance:
uuid: "83cc342659354e3a88460ab25db335a4"
format_stamp: "Formatted at 2026-09-26 19:44:41 on dist-test-slave-mrw4"
server_key: "c396bb8e28c4a6c66ca36fa70345475b"
server_key_iv: "05c7b8a8478f766270375ccd4b3c3d04"
server_key_version: "encryptionkey@0"
I20260926 19:44:41.636008 25503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/wal/instance:
uuid: "83cc342659354e3a88460ab25db335a4"
format_stamp: "Formatted at 2026-09-26 19:44:41 on dist-test-slave-mrw4"
server_key: "c396bb8e28c4a6c66ca36fa70345475b"
server_key_iv: "05c7b8a8478f766270375ccd4b3c3d04"
server_key_version: "encryptionkey@0"
I20260926 19:44:41.640727 25503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:44:41.643889 25518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:41.645088 25503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:44:41.645279 25503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/wal
uuid: "83cc342659354e3a88460ab25db335a4"
format_stamp: "Formatted at 2026-09-26 19:44:41 on dist-test-slave-mrw4"
server_key: "c396bb8e28c4a6c66ca36fa70345475b"
server_key_iv: "05c7b8a8478f766270375ccd4b3c3d04"
server_key_version: "encryptionkey@0"
I20260926 19:44:41.645467 25503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:41.670141 25503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:41.673859 25503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:41.674078 25503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:41.690029 25503 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.125:34627
I20260926 19:44:41.690099 25569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.125:34627 every 8 connection(s)
I20260926 19:44:41.691242 25503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/data/info.pb
I20260926 19:44:41.694718 25570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:41.700443 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 25503
I20260926 19:44:41.700605 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-1/wal/instance
I20260926 19:44:41.701014 2705 external_mini_cluster.cc:1428] Setting key e9bc91a402ee8cec4689458d296f6d71
I20260926 19:44:41.700758 25570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:44:41 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451881, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:44:41.720479 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.703176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.2.164.126:41673)
0926 19:44:41.703388 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:41.703393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:41.704557 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 19:44:41.706814 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:41.706818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:41.706892 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:41.706980 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:41.708763 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.709540 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.711095 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.711682 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.711808 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:41.715317 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:41.715322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:41.715326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:41.715363 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:41.717702 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:41.718487 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:41.718490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:41.718492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:41.718594 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:41.718965 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:41.718967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:41.718969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:41.719221 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:41.719323 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:41.720073 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 19:44:41.720246 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:44:41.721832 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.703669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41673 (local address 127.0.0.1:60148)
0926 19:44:41.704386 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.704429 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:41.706511 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:41.707223 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:41.707234 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:41.707901 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:41.708573 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:41.708596 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.709691 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.709696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:41.710948 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:41.710960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.711931 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.711938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:41.712141 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:41.712794 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:41.712816 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:41.715100 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:41.717828 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:41.717837 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:41.717852 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:41.718396 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:41.718676 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:41.718680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:41.718683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:41.718879 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:41.719326 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:41.719333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:41.719713 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 19:44:41.720009 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 19:44:41.720503 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":269,"threads_started":1}
I20260926 19:44:41.723476 25570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.164.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.164.124@KRBTEST.COM" created.
Sep 26 19:44:41 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451881, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
I20260926 19:44:41.736501 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.724242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36800)
0926 19:44:41.724437 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.724455 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:41.724562 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:41.725272 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:41.725276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:41.725667 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:41.725784 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:41.725792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.727076 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.727081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:41.727877 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:41.727890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.728615 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.728621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:41.728791 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:41.729351 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:41.729366 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:41.731300 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:41.734129 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:41.734134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:41.734137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:41.734592 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:41.734903 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:41.734907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:41.734911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:41.735074 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:41.735634 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:41.735638 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:41.735774 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 19:44:41.735895 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 19:44:41.736139 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 19:44:41.736605 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.724344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.2.164.125:34627)
0926 19:44:41.724881 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:41.724887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:41.724911 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:44:41.724948 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:41.724953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:41.725122 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:41.725276 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:41.725942 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.726956 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.727991 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.728492 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.728594 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:41.731491 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:41.731501 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:41.731514 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:41.731541 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:41.734008 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:41.734702 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:41.734708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:41.734711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:41.734815 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:41.735166 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:41.735171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:41.735174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:41.735498 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:41.735701 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:41.736042 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 19:44:41.736255 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":253,"threads_started":1}
I20260926 19:44:41.738930 25570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:44:41.740011 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.739602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:32988)
0926 19:44:41.739747 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.739873 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 19:44:41.740252 25520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
I20260926 19:44:41.741645 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.741359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33004)
0926 19:44:41.741486 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.741556 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 19:44:41.741825 25570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:39719: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.164.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab.
Entry for principal kudu/127.2.164.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab.
I20260926 19:44:41.757032 2705 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
/tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.124:39719
--webserver_interface=127.2.164.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab
--principal=kudu/127.2.164.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.164.84:45471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.164.126:41673,127.2.164.125:34627,127.2.164.124:39719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:44:41.791083 25570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } attempt: 1
I20260926 19:44:41.792173 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.791738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33012)
0926 19:44:41.791927 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.792031 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 19:44:41.793270 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.792872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33024)
0926 19:44:41.793021 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.793111 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 19:44:41.793470 25570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:39719: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
I20260926 19:44:41.861682 25570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } attempt: 2
I20260926 19:44:41.862691 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.862262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33036)
0926 19:44:41.862448 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.862541 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 19:44:41.863736 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.863383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33038)
0926 19:44:41.863524 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.863589 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 19:44:41.863926 25570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:39719: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
I20260926 19:44:41.917841 25496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } attempt: 5
Sep 26 19:44:41 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451881, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
I20260926 19:44:41.931092 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.918661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36802)
0926 19:44:41.918859 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.918880 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:41.919020 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:41.919375 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:41.919380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:41.919761 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:41.919944 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:41.919954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.920922 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.920928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:41.921839 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:41.921853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.922592 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.922598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:41.922785 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:41.923350 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:41.923376 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:41.925764 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:41.928587 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:41.928593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:41.928597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:41.929108 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:41.929536 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:41.929541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:41.929545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:41.929682 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:41.930178 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:41.930183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:41.930325 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 19:44:41.930462 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 19:44:41.930728 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 19:44:41.931103 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.918801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.2.164.125:34627)
0926 19:44:41.919012 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:41.919017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:41.919040 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:44:41.919127 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:41.919132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:41.919218 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:41.919303 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:41.920105 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.920785 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.921996 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:41.922446 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:41.922568 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:41.925976 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:41.925981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:41.925984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:41.926017 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:41.928443 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:41.929301 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:41.929306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:41.929309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:41.929432 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:41.929789 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:41.929794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:41.929797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:41.930068 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:41.930197 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:41.930590 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 19:44:41.930803 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 19:44:41.932148 25496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:44:41.933271 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.932887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33040)
0926 19:44:41.933042 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.933109 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 19:44:41.934163 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.933866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33052)
0926 19:44:41.933989 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.934072 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 19:44:41.934382 25496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:39719: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
I20260926 19:44:41.955120 25570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } attempt: 3
I20260926 19:44:41.956094 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.955654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33060)
0926 19:44:41.955850 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.955944 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 19:44:41.957149 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.956751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33064)
0926 19:44:41.956965 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.957039 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260926 19:44:41.957405 25570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:39719: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
I20260926 19:44:41.985575 25496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } attempt: 1
I20260926 19:44:41.986640 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.986197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33074)
0926 19:44:41.986401 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.986487 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 19:44:41.987521 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:41.987212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33076)
0926 19:44:41.987349 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:41.987418 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 19:44:41.987702 25496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:39719: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
I20260926 19:44:42.055914 25496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } attempt: 2
I20260926 19:44:42.056988 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.056537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33084)
0926 19:44:42.056736 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:42.056819 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 19:44:42.057899 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.057567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33090)
0926 19:44:42.057722 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:42.057794 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 19:44:42.058104 25496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:39719: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
W20260926 19:44:42.065256 25577 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:44:42.065555 25577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:44:42.065646 25577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:44:42.065762 25577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:44:42.072865 25577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:44:42.072952 25577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:44:42.073041 25577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:44:42.073102 25577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:44:42.073201 25577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:44:42.082677 25577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.164.84:45471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.164.126:41673,127.2.164.125:34627,127.2.164.124:39719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/krb5kdc/kudu_127.2.164.124.keytab
--principal=kudu/127.2.164.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.164.124:39719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.164.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:44:42.083935 25577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:44:42.085126 25577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:44:42.095793 25582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:42.095737 25585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:44:42.095746 25583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:44:42.096710 25577 server_base.cc:1034] running on GCE node
I20260926 19:44:42.097464 25577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:44:42.098702 25577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:44:42.099912 25577 hybrid_clock.cc:648] HybridClock initialized: now 1790451882099838 us; error 44 us; skew 500 ppm
I20260926 19:44:42.100196 25577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:44:42.100580 25570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } attempt: 4
I20260926 19:44:42.101568 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.101122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33104)
0926 19:44:42.101341 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:42.101425 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 19:44:42.102536 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.102234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33114)
0926 19:44:42.102364 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:42.102434 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 19:44:42.102733 25570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.164.124:39719: Network error: Client connection negotiation failed: client connection to 127.2.164.124:39719: connect: Connection refused (error 111)
Sep 26 19:44:42 dist-test-slave-mrw4 krb5kdc[25416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451882, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:44:42.103535 25577 init.cc:377] Logged in from keytab as kudu/127.2.164.124@KRBTEST.COM (short username kudu)
I20260926 19:44:42.104853 25577 webserver.cc:460] Webserver started at http://127.2.164.124:43819/ using document root <none> and password file <none>
I20260926 19:44:42.105511 25577 fs_manager.cc:359] Metadata directory not provided
I20260926 19:44:42.105605 25577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:44:42.105888 25577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:44:42.108557 25577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/data/instance:
uuid: "0f466c468b69430cb9e260a9fdcadb1f"
format_stamp: "Formatted at 2026-09-26 19:44:42 on dist-test-slave-mrw4"
server_key: "cf557c01a052841dc383074382910137"
server_key_iv: "c6dd3ac5b941e99df195d17bca83298e"
server_key_version: "encryptionkey@0"
I20260926 19:44:42.109330 25577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/wal/instance:
uuid: "0f466c468b69430cb9e260a9fdcadb1f"
format_stamp: "Formatted at 2026-09-26 19:44:42 on dist-test-slave-mrw4"
server_key: "cf557c01a052841dc383074382910137"
server_key_iv: "c6dd3ac5b941e99df195d17bca83298e"
server_key_version: "encryptionkey@0"
I20260926 19:44:42.113891 25577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 19:44:42.116870 25592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:44:42.118130 25577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 19:44:42.118279 25577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/data,/tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/wal
uuid: "0f466c468b69430cb9e260a9fdcadb1f"
format_stamp: "Formatted at 2026-09-26 19:44:42 on dist-test-slave-mrw4"
server_key: "cf557c01a052841dc383074382910137"
server_key_iv: "c6dd3ac5b941e99df195d17bca83298e"
server_key_version: "encryptionkey@0"
I20260926 19:44:42.118458 25577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:44:42.136758 25577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:44:42.140547 25577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:44:42.140774 25577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:44:42.156898 25577 rpc_server.cc:307] RPC server started. Bound to: 127.2.164.124:39719
I20260926 19:44:42.156919 25643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.164.124:39719 every 8 connection(s)
I20260926 19:44:42.158133 25577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/data/info.pb
I20260926 19:44:42.161515 25644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:44:42.164297 25496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } attempt: 3
I20260926 19:44:42.168154 2705 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu as pid 25577
I20260926 19:44:42.168336 2705 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDi4fXZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790451540687502-2705-0/minicluster-data/master-2/wal/instance
I20260926 19:44:42.168859 2705 external_mini_cluster.cc:1428] Setting key e57f562b8a78ae37e9a92d69a8bb2b1d
I20260926 19:44:42.168288 25644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:44:42 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451881, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.126@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
I20260926 19:44:42.186214 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.164904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33124)
0926 19:44:42.165142 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:42.165187 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:42.165318 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:42.168865 (+ 3547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:42.168870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:42.169386 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:42.169586 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:42.169600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.172080 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.172087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:42.173663 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:42.173681 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.175117 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.175124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:42.175333 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.175958 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:42.175984 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:42.178861 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:42.183187 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:42.183192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:42.183196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:42.183711 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:42.184213 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:42.184219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:42.184223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:42.184383 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:42.185220 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:42.185225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:42.185392 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 19:44:42.185528 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 19:44:42.185795 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 19:44:42.187757 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.165746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.2.164.124:39719)
0926 19:44:42.166474 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.166485 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.166522 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:44:42.168072 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.168093 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.168543 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.168881 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.169772 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.171874 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.173878 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.174938 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.175693 (+ 755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.179236 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:42.179246 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:42.179266 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:42.179311 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:42.183002 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.183869 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.183881 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.183889 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.184093 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.184510 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.184515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.184519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.185034 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:42.185257 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:42.185788 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 19:44:42.186207 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":315,"threads_started":1}
Sep 26 19:44:42 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451882, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
Sep 26 19:44:42 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
I20260926 19:44:42.194875 25496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809: Bootstrap starting.
I20260926 19:44:42.196139 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.171870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.2.164.126:41673)
0926 19:44:42.172067 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.172073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.181071 (+ 8998us) server_negotiation.cc:407] Connection header received
0926 19:44:42.181380 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.181385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.181462 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.181530 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.182823 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.183600 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.185470 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.186144 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.186270 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.190018 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:42.190023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:42.190027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:42.190062 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:42.193151 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.193946 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.193951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.193954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.194122 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.194557 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.194562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.194565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.194903 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:42.195035 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:42.195560 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 19:44:42.195782 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:44:42.196357 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.171334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.126:41673 (local address 127.0.0.1:60152)
0926 19:44:42.180941 (+ 9607us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:42.180983 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:42.181206 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:42.181735 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:42.181742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:42.182335 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:42.182634 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:42.182643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.183742 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.183747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:42.185332 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:42.185354 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.186481 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.186487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:42.186637 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.187165 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:42.187186 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:42.189855 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:42.193294 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:42.193304 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:42.193310 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:42.193811 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:42.194243 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:42.194248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:42.194251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:42.194443 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:42.195029 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:42.195036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:42.195260 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 19:44:42.195415 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 19:44:42.195969 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9360,"thread_start_us":9217,"threads_started":1}
I20260926 19:44:42.197846 25644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:44:42.198773 25496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:42.200053 25496 log.cc:824] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:42.200157 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.169656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.2.164.124:39719)
0926 19:44:42.181635 (+ 11979us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.181643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.182560 (+ 917us) server_negotiation.cc:407] Connection header received
0926 19:44:42.182716 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.182725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.182798 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.182894 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.184501 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.185781 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.187539 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.188175 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.188350 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.193812 (+ 5462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:42.193817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:42.193820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:42.193847 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:42.197351 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.198110 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.198115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.198118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.198243 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.198587 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.198592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.198596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.198873 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:42.199285 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:42.199573 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 19:44:42.199774 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11858,"thread_start_us":11733,"threads_started":1}
Sep 26 19:44:42 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451882, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
I20260926 19:44:42.208045 25496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809: No bootstrap required, opened a new log
Sep 26 19:44:42 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.126@KRBTEST.COM
I20260926 19:44:42.213779 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.198541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.125:34627 (local address 127.0.0.1:36818)
0926 19:44:42.199076 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:42.199095 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:42.199253 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:42.199734 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:42.199752 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:42.200187 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:42.200378 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:42.200389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.201609 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.201615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:42.203067 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:42.203081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.203998 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.204005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:42.204193 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.204765 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:42.204787 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:42.207321 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:42.210941 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:42.210946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:42.210950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:42.211538 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:42.211984 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:42.211989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:42.211992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:42.212146 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:42.212842 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:42.212846 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:42.213004 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 19:44:42.213144 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 19:44:42.213441 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430}
I20260926 19:44:42.214334 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.198900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.2.164.125:34627)
0926 19:44:42.199063 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.199068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.199157 (+ 89us) server_negotiation.cc:407] Connection header received
0926 19:44:42.199401 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.199406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.199489 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.199572 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.200503 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.201481 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.203225 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.203847 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.203970 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.207534 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:42.207542 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:42.207546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:42.207587 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:42.210789 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.211669 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.211675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.211678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.211842 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.212284 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.212289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.212293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.212620 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:42.212746 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:42.213300 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 19:44:42.213521 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:44:42.215150 25644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:44:42.218041 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.201934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.2.164.126:41673)
0926 19:44:42.202075 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.202078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.202091 (+ 13us) server_negotiation.cc:407] Connection header received
0926 19:44:42.202127 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.202130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.202180 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.202266 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.203454 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.204437 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.205976 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.206461 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.206571 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.210419 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:42.210423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:42.210429 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:42.210458 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:42.212930 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.213657 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.213662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.213666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.214151 (+ 485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.214597 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.214603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.214606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.214967 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:42.215270 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:42.215550 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 19:44:42.215777 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 19:44:42.219352 25496 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } }
I20260926 19:44:42.219750 25496 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4bcaba08bd466aa223499cf2334809, State: Initialized, Role: FOLLOWER
I20260926 19:44:42.220247 25496 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } }
Sep 26 19:44:42 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451882, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.124@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
I20260926 19:44:42.228153 25651 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } } }
I20260926 19:44:42.228540 25651 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 19:44:42 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.164.125@KRBTEST.COM
I20260926 19:44:42.230295 25496 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:42.231119 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.216103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.2.164.124:39719)
0926 19:44:42.216281 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.216286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.216444 (+ 158us) server_negotiation.cc:407] Connection header received
0926 19:44:42.216569 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.216574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.216630 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.216698 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.217379 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.218043 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.219617 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.220315 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.220405 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.224497 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:42.224503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:42.224508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:42.224540 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:42.228029 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.228926 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.228931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.228934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.229067 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.229476 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.229480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.229483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.229857 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:42.229974 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:42.230588 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 19:44:42.230796 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 19:44:42.232102 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.215806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33146)
0926 19:44:42.216374 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:42.216388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:42.216480 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:42.216725 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:42.216729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:42.217077 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:42.217249 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:42.217260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.218317 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.218321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:42.219491 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:42.219505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.220619 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.220625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:42.220805 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.221622 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:42.221645 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:42.224338 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:42.228181 (+ 3843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:42.228185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:42.228189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:42.228798 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:42.229220 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:42.229224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:42.229227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:42.229369 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:42.230114 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:42.230118 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:42.230257 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 19:44:42.231521 (+ 1264us) client_negotiation.cc:240] Negotiation successful
0926 19:44:42.231787 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392}
I20260926 19:44:42.236375 25644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f: Bootstrap starting.
I20260926 19:44:42.239910 25644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:42.241178 25644 log.cc:824] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f: Log is configured to *not* fsync() on all Append() calls
W20260926 19:44:42.241277 25662 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:44:42.241401 25662 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:44:42.245596 25644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f: No bootstrap required, opened a new log
I20260926 19:44:42.246939 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.220053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.2.164.125:34627)
0926 19:44:42.220206 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.220212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.220426 (+ 214us) server_negotiation.cc:407] Connection header received
0926 19:44:42.220641 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.220647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.220719 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.220783 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.222491 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.223516 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.225304 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.225956 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.226049 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.231094 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:42.231100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:42.231104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:42.231143 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:42.243133 (+ 11990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.244017 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.244023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.244027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.244193 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.244672 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.244677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.244680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.245018 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:42.245334 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:42.246374 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0926 19:44:42.246612 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:44:42.251590 2705 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:44:42.254602 25471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bc4bcaba08bd466aa223499cf2334809"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60162
I20260926 19:44:42.241678 25429 master_runner.cc:186] Error getting master registration for 127.2.164.125:34627: OK, instance_id { permanent_uuid: "83cc342659354e3a88460ab25db335a4" instance_seqno: 1790451881685504 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:44:42.254992 25471 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 19:44:42.255018 25429 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 19:44:42.255230 25471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:42.255570 25644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } }
I20260926 19:44:42.255980 25644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f466c468b69430cb9e260a9fdcadb1f, State: Initialized, Role: FOLLOWER
I20260926 19:44:42.256477 25644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } }
I20260926 19:44:42.257424 25665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } } }
I20260926 19:44:42.257874 25665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:42.258589 25644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:42.258649 25471 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } }
I20260926 19:44:42.260581 25471 leader_election.cc:288] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [CANDIDATE]: Term 1 election: Requested vote from peers 83cc342659354e3a88460ab25db335a4 (127.2.164.125:34627), 0f466c468b69430cb9e260a9fdcadb1f (127.2.164.124:39719)
I20260926 19:44:42.261303 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc4bcaba08bd466aa223499cf2334809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0f466c468b69430cb9e260a9fdcadb1f"
I20260926 19:44:42.261807 25619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:42.265240 25619 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4bcaba08bd466aa223499cf2334809 in term 1.
I20260926 19:44:42.265856 25446 leader_election.cc:302] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f466c468b69430cb9e260a9fdcadb1f, bc4bcaba08bd466aa223499cf2334809; no voters:
I20260926 19:44:42.266270 25651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:44:42.266873 25651 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [term 1 LEADER]: Becoming Leader. State: Replica: bc4bcaba08bd466aa223499cf2334809, State: Running, Role: LEADER
I20260926 19:44:42.267400 25651 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } }
I20260926 19:44:42.270767 25671 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc4bcaba08bd466aa223499cf2334809. Latest consensus state: current_term: 1 leader_uuid: "bc4bcaba08bd466aa223499cf2334809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } } }
I20260926 19:44:42.271157 25671 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:42.281996 25676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:44:42.285656 25676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:44:42.300838 25684 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 19:44:42.308384 25619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [term 1 FOLLOWER]: Refusing update from remote peer bc4bcaba08bd466aa223499cf2334809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:44:42.309638 25651 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:44:42.323536 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.296122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.2.164.125:34627)
0926 19:44:42.296314 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.296321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.307021 (+ 10700us) server_negotiation.cc:407] Connection header received
0926 19:44:42.307285 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.307292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.307374 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.307445 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.309821 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.310973 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.312819 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.313480 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.313575 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.317004 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:42.317011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:42.317015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:42.317054 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:42.320390 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.321149 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.321154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.321158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.321331 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.321802 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.321807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.321810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.322177 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:42.322390 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:42.322975 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 19:44:42.323195 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:44:42.307610 25577 master_runner.cc:186] Error getting master registration for 127.2.164.125:34627: OK, instance_id { permanent_uuid: "83cc342659354e3a88460ab25db335a4" instance_seqno: 1790451881685504 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:44:42.329511 25577 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 19:44:42.339787 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.302806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.2.164.124:39719)
0926 19:44:42.302994 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.303000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.311975 (+ 8975us) server_negotiation.cc:407] Connection header received
0926 19:44:42.312464 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.312470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.312544 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.312617 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.317206 (+ 4589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.318286 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.322979 (+ 4693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.323674 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.323765 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.331123 (+ 7358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:42.331129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:42.331134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:42.331197 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:42.334588 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.335360 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.335366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.335369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.335654 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.336434 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.336471 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.336475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.337200 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:42.337696 (+ 496us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:42.338690 (+ 994us) server_negotiation.cc:299] Negotiation successful
0926 19:44:42.339100 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:44:42.340447 25665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc4bcaba08bd466aa223499cf2334809. Latest consensus state: current_term: 1 leader_uuid: "bc4bcaba08bd466aa223499cf2334809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } } }
I20260926 19:44:42.340715 25665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:42.342449 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.308685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.2.164.126:41673)
0926 19:44:42.308841 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.308846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.315018 (+ 6172us) server_negotiation.cc:407] Connection header received
0926 19:44:42.315261 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.315266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.315342 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.315420 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.320222 (+ 4802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.321315 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.325215 (+ 3900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.325917 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.326006 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.335394 (+ 9388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:42.335401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:42.335406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:42.335446 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:42.338913 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.339681 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.339686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.339689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.339859 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.340279 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.340284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.340287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.340801 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:42.341024 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:42.341713 (+ 689us) server_negotiation.cc:299] Negotiation successful
0926 19:44:42.342102 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:44:42.344471 25671 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc4bcaba08bd466aa223499cf2334809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } } }
I20260926 19:44:42.344723 25671 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [sys.catalog]: This master's current role is: LEADER
I20260926 19:44:42.346450 25676 catalog_manager.cc:1347] Generated new cluster ID: e1993872f7d742dba82e05a19aa72916
I20260926 19:44:42.346541 25676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:44:42.369737 25665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc4bcaba08bd466aa223499cf2334809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } } }
I20260926 19:44:42.369987 25665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:42.375525 25676 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:44:42.378031 25676 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:44:42.401923 25570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } attempt: 5
I20260926 19:44:42.402726 25676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809: Generated new TSK 0
I20260926 19:44:42.403822 25676 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 26 19:44:42 dist-test-slave-mrw4 krb5kdc[25416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451881, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.164.125@KRBTEST.COM for kudu/127.2.164.124@KRBTEST.COM
I20260926 19:44:42.417788 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.402496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.2.164.124:39719)
0926 19:44:42.403006 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.403011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.403028 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:44:42.403073 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.403077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.403142 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.403230 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.404218 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.405364 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.407017 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.407703 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.407822 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.411215 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:44:42.411221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:44:42.411226 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:44:42.411271 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:44:42.415011 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.415823 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.415828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.415832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.415971 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:44:42.416417 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:44:42.416423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:44:42.416426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:44:42.416745 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:44:42.416913 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:42.417268 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 19:44:42.417458 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391}
I20260926 19:44:42.417799 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:42.402441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.164.124:39719 (local address 127.0.0.1:33154)
0926 19:44:42.402660 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 19:44:42.402679 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:44:42.402786 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:44:42.403297 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:44:42.403303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:44:42.403795 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:44:42.403997 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:42.404011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.405491 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.405497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:42.406885 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:44:42.406899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:44:42.407838 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:44:42.407844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:44:42.408070 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:42.408661 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:44:42.408677 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:44:42.411038 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:44:42.415238 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:42.415244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:42.415248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:42.415725 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:42.416138 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:44:42.416144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:44:42.416147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:44:42.416299 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:44:42.416877 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:44:42.416882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:44:42.417029 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 19:44:42.417148 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 19:44:42.417449 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 19:44:42.423337 25570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4: Bootstrap starting.
I20260926 19:44:42.427110 25570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4: Neither blocks nor log segments found. Creating new log.
I20260926 19:44:42.428323 25570 log.cc:824] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4: Log is configured to *not* fsync() on all Append() calls
I20260926 19:44:42.437674 25570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4: No bootstrap required, opened a new log
I20260926 19:44:42.448318 25570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } }
I20260926 19:44:42.448700 25570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83cc342659354e3a88460ab25db335a4, State: Initialized, Role: FOLLOWER
I20260926 19:44:42.449294 25570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } }
I20260926 19:44:42.450382 25694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } } }
I20260926 19:44:42.450870 25694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:42.451368 25570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:44:42.456836 25545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc4bcaba08bd466aa223499cf2334809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "83cc342659354e3a88460ab25db335a4"
I20260926 19:44:42.457365 25545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:44:42.460958 25545 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4bcaba08bd466aa223499cf2334809 in term 1.
I20260926 19:44:42.462594 25544 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [term 1 FOLLOWER]: Refusing update from remote peer bc4bcaba08bd466aa223499cf2334809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:44:42.463544 25671 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:44:42.500998 25694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc4bcaba08bd466aa223499cf2334809. Latest consensus state: current_term: 1 leader_uuid: "bc4bcaba08bd466aa223499cf2334809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } } }
I20260926 19:44:42.501366 25694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:44:42.501806 25706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc4bcaba08bd466aa223499cf2334809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } } }
I20260926 19:44:42.502239 25706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 19:44:42.513840 25574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.505001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.2.164.125:34627)
0926 19:44:42.505221 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.505227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.511959 (+ 6732us) server_negotiation.cc:407] Connection header received
0926 19:44:42.512134 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.512140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.512246 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.512339 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.513595 (+ 1256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36834: BlockingRecv error: recv got EOF from 127.0.0.1:36834 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:44:42.513607 25717 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 19:44:42.515733 25498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.504615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.2.164.126:41673)
0926 19:44:42.504832 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.504838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.514139 (+ 9301us) server_negotiation.cc:407] Connection header received
0926 19:44:42.514414 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.514420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.514504 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.514579 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.515509 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60178: BlockingRecv error: recv got EOF from 127.0.0.1:60178 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 19:44:42.518020 25647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.511647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.2.164.124:39719)
0926 19:44:42.511812 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.511818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.516517 (+ 4699us) server_negotiation.cc:407] Connection header received
0926 19:44:42.516753 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.516758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.516836 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.516906 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.517771 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33164: BlockingRecv error: recv got EOF from 127.0.0.1:33164 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 19:44:42.527774 25574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.526089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.2.164.125:34627)
0926 19:44:42.526223 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.526228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.526573 (+ 345us) server_negotiation.cc:407] Connection header received
0926 19:44:42.526758 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.526763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.526848 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.526929 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.527534 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36838: BlockingRecv error: recv got EOF from 127.0.0.1:36838 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 19:44:42.532886 25574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.531645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.2.164.125:34627)
0926 19:44:42.531825 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.531830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.531851 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:44:42.531965 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.531970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.532026 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.532098 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.532716 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36840: BlockingRecv error: recv got EOF from 127.0.0.1:36840 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 19:44:42.542749 25574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.537477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.2.164.125:34627)
0926 19:44:42.537597 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.537602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.541612 (+ 4010us) server_negotiation.cc:407] Connection header received
0926 19:44:42.541768 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.541772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.541844 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.541933 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.542539 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36844: BlockingRecv error: recv got EOF from 127.0.0.1:36844 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 19:44:42.551116 25574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.549480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.2.164.125:34627)
0926 19:44:42.549663 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.549668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.549952 (+ 284us) server_negotiation.cc:407] Connection header received
0926 19:44:42.550114 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.550119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.550187 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.550263 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.550919 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36856: BlockingRecv error: recv got EOF from 127.0.0.1:36856 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 19:44:42.564033 25574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.562480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.2.164.125:34627)
0926 19:44:42.562651 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.562657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.562954 (+ 297us) server_negotiation.cc:407] Connection header received
0926 19:44:42.563085 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.563090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.563171 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.563290 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.563828 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36870: BlockingRecv error: recv got EOF from 127.0.0.1:36870 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 19:44:42.584753 25574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.582952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.2.164.125:34627)
0926 19:44:42.583149 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.583154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.583435 (+ 281us) server_negotiation.cc:407] Connection header received
0926 19:44:42.583602 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.583608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.583711 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.583833 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.584527 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36882: BlockingRecv error: recv got EOF from 127.0.0.1:36882 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 19:44:42.621739 25574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.619838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.2.164.125:34627)
0926 19:44:42.620043 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.620050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.620312 (+ 262us) server_negotiation.cc:407] Connection header received
0926 19:44:42.620497 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.620503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.620613 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.620734 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.621480 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36888: BlockingRecv error: recv got EOF from 127.0.0.1:36888 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 19:44:42.691222 25574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.688787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36902 (local address 127.2.164.125:34627)
0926 19:44:42.688973 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.688980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.689604 (+ 624us) server_negotiation.cc:407] Connection header received
0926 19:44:42.689878 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.689884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.689983 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.690092 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.691011 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36902: BlockingRecv error: recv got EOF from 127.0.0.1:36902 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 19:44:42.824932 25574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:42.823118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.2.164.125:34627)
0926 19:44:42.823284 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:42.823289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:42.823690 (+ 401us) server_negotiation.cc:407] Connection header received
0926 19:44:42.823878 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:42.823882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:42.823954 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:42.824043 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:42.824695 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36912: BlockingRecv error: recv got EOF from 127.0.0.1:36912 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 19:44:42.898801 25671 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc4bcaba08bd466aa223499cf2334809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4bcaba08bd466aa223499cf2334809" member_type: VOTER last_known_addr { host: "127.2.164.126" port: 41673 } } peers { permanent_uuid: "83cc342659354e3a88460ab25db335a4" member_type: VOTER last_known_addr { host: "127.2.164.125" port: 34627 } } peers { permanent_uuid: "0f466c468b69430cb9e260a9fdcadb1f" member_type: VOTER last_known_addr { host: "127.2.164.124" port: 39719 } } }
I20260926 19:44:42.899107 25671 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4bcaba08bd466aa223499cf2334809 [sys.catalog]: This master's current role is: LEADER
W20260926 19:44:43.088527 25574 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:44:43.086829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.2.164.125:34627)
0926 19:44:43.087044 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:43.087050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:43.087197 (+ 147us) server_negotiation.cc:407] Connection header received
0926 19:44:43.087416 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:43.087420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:43.087492 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:43.087579 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:44:43.088311 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36914: BlockingRecv error: recv got EOF from 127.0.0.1:36914 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:44:43.305097 25684 catalog_manager.cc:1259] Loaded cluster ID: e1993872f7d742dba82e05a19aa72916
I20260926 19:44:43.305238 25684 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f: loading cluster ID for follower catalog manager: success
I20260926 19:44:43.307688 25684 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f: acquiring CA information for follower catalog manager: success
I20260926 19:44:43.309511 25684 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f466c468b69430cb9e260a9fdcadb1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:44:43.519755 25717 catalog_manager.cc:1259] Loaded cluster ID: e1993872f7d742dba82e05a19aa72916
I20260926 19:44:43.519973 25717 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4: loading cluster ID for follower catalog manager: success
I20260926 19:44:43.523800 25717 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4: acquiring CA information for follower catalog manager: success
I20260926 19:44:43.526263 25717 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 83cc342659354e3a88460ab25db335a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:44:43.624079 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:43.604138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.2.164.125:34627)
0926 19:44:43.616994 (+ 12856us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:43.617002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:43.617028 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:44:43.617109 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:43.617114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:43.617341 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:43.617484 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:44:43.617976 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:43.619185 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:43.621127 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:43.621810 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:43.621941 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:43.622582 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:44:43.623357 (+ 775us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:44:43.623507 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:43.623634 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 19:44:43.623723 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12119,"thread_start_us":11970,"threads_started":1}
I20260926 19:44:43.649456 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:43.634537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.2.164.124:39719)
0926 19:44:43.642667 (+ 8130us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:43.642673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:43.642692 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:44:43.642793 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:43.642799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:43.642961 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:43.643116 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:44:43.643616 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:43.644779 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:43.646549 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:43.647213 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:43.647333 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:43.647867 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:44:43.648667 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:44:43.648802 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:43.648918 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 19:44:43.648999 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8006,"thread_start_us":7875,"threads_started":1}
I20260926 19:44:43.676059 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:43.655841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.2.164.125:34627)
0926 19:44:43.656140 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:43.656147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:43.669197 (+ 13050us) server_negotiation.cc:407] Connection header received
0926 19:44:43.669509 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:43.669515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:43.669602 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:43.669664 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:44:43.670330 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:43.671535 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:43.673684 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:43.674514 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:43.674608 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:43.675280 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:44:43.675424 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:44:43.675485 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:43.675702 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 19:44:43.675782 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260926 19:44:43.678582 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:43.655681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.2.164.126:41673)
0926 19:44:43.669703 (+ 14022us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:43.669710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:43.670668 (+ 958us) server_negotiation.cc:407] Connection header received
0926 19:44:43.670814 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:43.670819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:43.671030 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:43.671203 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:44:43.671957 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:43.673110 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:43.675413 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:43.676079 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:43.676433 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:43.677180 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:44:43.677657 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:44:43.677760 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:43.678161 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 19:44:43.678254 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13873,"thread_start_us":14867,"threads_started":1}
I20260926 19:44:43.683015 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 25429
I20260926 19:44:43.683967 25491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:43.684206 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:44:43.655939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.2.164.124:39719)
0926 19:44:43.656084 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 19:44:43.656090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:44:43.678000 (+ 21910us) server_negotiation.cc:407] Connection header received
0926 19:44:43.678231 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:44:43.678236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:44:43.678317 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:44:43.678406 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:44:43.678943 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:43.680004 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:43.682429 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:44:43.682983 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:44:43.683108 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:44:43.683545 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:44:43.683661 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:44:43.683744 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 19:44:43.683871 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 19:44:43.683941 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 19:44:43.802595 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 25429
I20260926 19:44:43.824736 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 25503
I20260926 19:44:43.825907 25565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:43.938437 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 25503
I20260926 19:44:43.953935 2705 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu pid 25577
I20260926 19:44:43.955030 25639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:44:44.059353 2705 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDi4fXZ/build/asan/bin/kudu with pid 25577
2026-09-26T19:44:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4470 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9176 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (343338 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST